Automatic bean placing system and method

Abstract

The invention relates to the field of image processing technology, and discloses an automatic bean placing system and method, wherein an automatic bean placing system comprises a soaking module; a data acquisition module; a data processing module; a prediction module, which is used to construct a full soaking prediction model, and use a training data set to train the full soaking prediction model, and use the full soaking prediction model to predict a new soybean full soaking time point; the invention collects historical data and trains a prediction model, and based on the predicted full soaking time data, collects image data of soybean soaking at the time point, and obtains the soybean soaking state after processing. If the state is satisfied, the bean placing process is directly performed, thereby effectively reducing the time of invalid soaking, thereby improving production efficiency and labor costs, realizing automatic, efficient and high-precision soybean soaking processing, and adjusting the setting of the threshold according to actual soaking requirements, and accurately predicting the time points of different soaking degrees.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to an automatic bean placing system and method. Background Art

[0002] Large-scale soy product factories consume a large amount of soybeans on a daily basis. The soybeans need to be soaked before processing. The soaking process is mostly done manually, and people need to judge the soaking status of the soybeans. Whether the soybeans are fully soaked directly affects the taste and output rate of subsequent soy products.

[0003] However, human judgment is mostly done by observing with the naked eye whether there are wrinkles on the soybean surface and taking samples for testing. This is not only time-consuming and labor-intensive, but also unable to accurately control the soaking time. Different soaking states often occur, resulting in the inability to maximize factory benefits. Summary of the invention

[0004] The purpose of the present invention is to provide an automatic bean placing system and method in order to solve the above problems.

[0005] The present invention provides an automatic bean placing method, comprising the following steps:

[0006] Step 100, obtaining multiple sets of soybean soaking video data;

[0007] Step 200, extracting image frames from each set of soybean soaking video data at a set interval, and preprocessing all the extracted image frames;

[0008] Step 300: Use the target detection algorithm to locate the soybean region in the image frame, segment the soybean region from the background, obtain a binary mask of the soybean region, separate individual soybeans from the binary mask of the soybean region using an image processing algorithm, and generate an independent binary mask for each soybean;

[0009] Step 400: Analyze each soybean binary mask, calculate the maximum projection of the soybean in the horizontal and vertical directions, obtain the pixel value of the grain length and grain width, convert the pixel value into a real length unit using the camera calibration parameter, and output the grain length and grain width data of all soybean grains in each image frame;

[0010] Step 500, setting a soaking swelling grain length threshold, a soaking swelling grain width threshold and a sufficient soaking judgment condition of soybeans, traversing all soybean grain length and grain width data in each image frame in chronological order, selecting an image frame of sufficient soaking after preliminary screening according to the sufficient soaking judgment condition, obtaining all image frames between the image frame of sufficient soaking after preliminary screening and the previous image frame from the soybean soaking video data, and performing pre-processing;

[0011] Step 600, processing all the image frames preprocessed in step 500 according to steps 300 to 400, and obtaining the grain length and grain width measurement results of all soybeans in all image frames between the image frame of the initial screening and the previous image frame, selecting the image frame data of accurate and sufficient soaking after traversing in time order, the image frame data of accurate and sufficient soaking includes the timestamp of the image frame and the grain length and grain width measurement data of all soybeans, and constructing a training data set based on the grain length and grain width measurement results of all soybeans in all image frames generated in step 400 and the image frame data of accurate and sufficient soaking;

[0012] Step 700: construct a sufficient foaming prediction model, and train the sufficient foaming prediction model using a training data set;

[0013] Specifically:

[0014] Based on the training data set, construct the input vector of the model;

[0015] Constructing a full foaming prediction model, the full foaming prediction model includes an input layer, a hidden layer and an output layer;

[0016] Step 800, set an acquisition time period, collect new soybean soaking image data at the set time interval, process the new soybean soaking image data according to steps 300 to 400 to generate an input vector, and input the input vector into a sufficient soaking prediction model, which outputs predicted sufficient soaking time data.

[0017] As a further optimization scheme of the present invention, image frames in each group of soybean soaking video data are extracted at a set interval time, and all the extracted image frames are preprocessed, specifically as follows:

[0018] Use the video processing library to read the soybean soaking video file;

[0019] Set the time interval for extracting image frames;

[0020] Traverse the video and use the read() function to extract the image frame at the set time point;

[0021] All extracted image frames are preprocessed.

[0022] As a further optimization scheme of the present invention, each soybean binary mask is analyzed, the maximum projection of the soybean in the horizontal and vertical directions is calculated, the pixel values ​​of the grain length and grain width are obtained, the pixel values ​​are converted into real length units using camera calibration parameters, and the grain length and grain width data of all soybean grains in each image frame are output;

[0023] Specifically:

[0024] The binary mask of each soybean was analyzed;

[0025] Calculate the projection of the mask in the horizontal direction and find the maximum projection width as the pixel value of the grain length;

[0026] Calculate the projection of the mask in the vertical direction and find the maximum projection height as the pixel value of the particle width;

[0027] Use the camera calibration parameters to calculate the conversion coefficient between pixels and actual length;

[0028] Multiply the pixel value by the conversion coefficient to obtain the actual length and width of the soybean grain;

[0029] For each image frame, the grain length and width measurement results of all soybeans in it are generated.

[0030] As a further optimization scheme of the present invention, the threshold value of the length of the soybean grain after soaking and swelling is 1.65 times of the initial length of the soybean grain, and the threshold value of the width of the soybean grain after soaking and swelling is 1.10 times of the initial soybean grain.

[0031] As a further optimization scheme of the present invention, the sufficient swelling judgment condition is that the grain length and grain width data of all soybeans in the image frame are greater than or equal to the soaking swelling grain length threshold and the soaking swelling grain width threshold.

[0032] As a further optimization solution of the present invention, the hidden layer has M neurons;

[0033] For the jth hidden layer neuron, its input is:

[0034] ;

[0035] in, is the weight from the i-th neuron in the input layer to the j-th neuron in the hidden layer;

[0036] is the input value of the i-th neuron in the input layer;

[0037] is the bias term of the jth hidden layer neuron;

[0038] The output of the hidden layer neurons is:

[0039] ;

[0040] Among them, f is the activation function.

[0041] As a further optimization solution of the present invention, the output layer has only one neuron, and the output value is the predicted foaming time;

[0042] The input of the output layer neurons is:

[0043] ;

[0044] is the weight from the jth neuron in the hidden layer to the neuron in the output layer;

[0045] is the output of the jth neuron in the hidden layer;

[0046] is the output layer bias term.

[0047] As a further optimization scheme of the present invention, during training, the difference between the predicted foaming time and the actual foaming time is used as a loss function, and the weight and bias are continuously adjusted using an optimization algorithm.

[0048] An automatic bean placing system, which is used to execute the automatic bean placing method as described above, comprises:

[0049] Soaking module, used for soaking soybeans and automatically placing beans;

[0050] A data acquisition module, used for acquiring multiple sets of video data of soybean soaking process;

[0051] The data processing module is used to process multiple sets of video data of soybean soaking process and form a training data set, which specifically includes the following steps:

[0052] Step S1, extracting image frames from each group of soybean soaking video data at a set interval time, and preprocessing all the extracted image frames;

[0053] Step S2, using a target detection algorithm to locate the soybean region in the image frame, segmenting the soybean region from the background, obtaining a binary mask of the soybean region, using an image processing algorithm to separate individual soybeans from the binary mask of the soybean region, and generating an independent binary mask for each soybean;

[0054] Step S3, analyzing each soybean binary mask, calculating the maximum projection of the soybean in the horizontal and vertical directions, obtaining the pixel values ​​of the grain length and grain width, converting the pixel values ​​into real length units using the camera calibration parameters, and outputting the grain length and grain width data of all soybean grains in each image frame;

[0055] Step S4, setting a soaking swelling grain length threshold, a soaking swelling grain width threshold and a sufficient soaking judgment condition of soybeans, traversing all soybean grain length and grain width data in each image frame in chronological order, selecting an image frame of sufficient soaking after preliminary screening according to the sufficient soaking judgment condition, obtaining all image frames between the image frame of sufficient soaking after preliminary screening and the previous image frame from the soybean soaking video data, and performing pre-processing;

[0056] Step S5, processing all the image frames preprocessed in step S4 according to steps S2 to S3, and obtaining the grain length and grain width measurement results of all soybeans in all image frames between the image frame of the initial screening and the previous image frame, selecting the image frame data of accurate and sufficient soaking after traversing in chronological order, the image frame data of accurate and sufficient soaking includes the timestamp of the image frame and the grain length and grain width measurement data of all soybeans, and constructing a training data set based on the grain length and grain width measurement results of all soybeans in all image frames generated in step S3 and the image frame data of accurate and sufficient soaking;

[0057] The prediction module is used to construct a sufficient soaking prediction model, and use a training data set to train the sufficient soaking prediction model, set an acquisition time period, collect new soybean soaking image data at the set time interval, process the new soybean soaking image data according to steps S2 to S3 to generate an input vector, and input it into the sufficient soaking prediction model, and the sufficient soaking prediction model outputs predicted sufficient soaking time data.

[0058] A storage medium stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the steps in the automatic bean placing method as described above can be performed.

[0059] The beneficial effects of the present invention are as follows: the present invention collects historical data and trains a prediction model, and based on the trained prediction model, collects new soybean soaking image data at a set time point, forms a new prediction data set and inputs the prediction model, thereby obtaining the predicted sufficient soaking time of the newly soaked soybeans, and based on the predicted sufficient soaking time data, collects soybean soaking image data at this time point, and the soybean soaking state can be obtained after processing. If it meets the conditions, the soybeans are directly put into the pot, thereby effectively reducing the time of invalid soaking, realizing precise soaking management, thereby improving production efficiency and labor costs, and realizing automated, efficient, and high-precision soybean soaking processing. The threshold setting can be adjusted according to actual soaking needs, and the time points of different soaking degrees can be accurately predicted. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of an automatic bean placing method of the present invention;

[0061] Figure 2 It is a module block diagram of an automatic bean placing system of the present invention. DETAILED DESCRIPTION

[0062] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only intended to enable those skilled in the art to better understand and implement the subject matter described herein. In addition, the features described relative to some examples may also be combined in other examples.

[0063] like Figure 1 As shown, an automatic bean placing method comprises the following steps:

[0064] Step 100, obtaining multiple sets of soybean soaking video data; the starting time point of all soybean soaking video data is the soybean soaking starting time point, and the ending time point is the time point when the soybeans are put into the soaking bucket, so that the video data time series includes the time point when the soybeans are soaked to the optimal soaking state;

[0065] Step 200, extracting image frames from each set of soybean soaking video data at a set interval, and preprocessing all the extracted image frames;

[0066] Specifically:

[0067] Use a video processing library (such as OpenCV's VideoCapture) to read the soybean soaking video file;

[0068] Set the time interval for extracting image frames, for example, extract one frame every 1 minute;

[0069] Traverse the video and use the read() function to extract the image frame at the set time point;

[0070] Pre-process all extracted image frames to improve the effect of subsequent processing;

[0071] Common preprocessing operations include:

[0072] Denoising: Use Gaussian filtering, median filtering and other algorithms to reduce noise;

[0073] Enhance contrast: Use algorithms such as histogram equalization or CLAHE to enhance image contrast;

[0074] Correct distortion: Use camera calibration parameters to correct image distortion;

[0075] Camera calibration parameters include: camera internal parameters (such as focal length, principal point coordinates, distortion coefficient, etc.) can be calibrated to correct the perspective and distortion effects in the image, ensuring a good linear correspondence between the projected size on the image and the actual size;

[0076] The camera external parameters (camera pose) can accurately calculate the relative position and pose of the camera and the target object (soybean), so that the pixel values ​​on the image can be corrected according to the known physical size;

[0077] Through the parameters obtained by calibration, the soybean pixel size measured on the image plane can be accurately converted into the length unit (such as centimeters or millimeters) in the real physical world;

[0078] Even when soybeans are soaked, the volume of soybeans expands, causing the distance between the camera and the surface of soybeans to change. As long as the distance information is recorded for each shot, the calibration parameters can be used to compensate for the effect of this distance change.

[0079] It should be noted that during the calibration process, the position and number of calibration plates need to be well controlled, and calibration should be performed at multiple angles and distances to obtain sufficiently accurate and robust calibration parameters;

[0080] Step 300: Use a target detection algorithm (such as FasterR-CNN, YOLO, etc.) to locate the soybean region in the image frame, segment the soybean region from the background, obtain a binary mask of the soybean region, use an image processing algorithm (such as corrosion, expansion, connected domain analysis, etc.) to separate individual soybeans from the binary mask of the soybean region, and generate an independent binary mask for each soybean;

[0081] Specifically:

[0082] Use object detection algorithms (such as FasterR-CNN, YOLO, etc.) to locate soybean areas in the image;

[0083] The target detection algorithm extracts image features through convolutional neural networks and uses prior box regression and classification to obtain the target's bounding box and category;

[0084] The output of an object detection algorithm is usually a list of multiple object bounding boxes and their confidence scores;

[0085] Example: In an image, the detection algorithm outputs 10 bounding boxes, 3 of which correspond to soybean areas and the rest are background;

[0086] Using the target detection results, the soybean area is segmented from the image;

[0087] Perform pixel-level segmentation on each soybean bounding box to generate a binary mask;

[0088] In the binary mask, the value of the pixel belonging to soybean is 1, and the value of the background pixel is 0;

[0089] The segmentation algorithm can use instance segmentation methods such as Grabcut and MaskR-CNN;

[0090] Example: For a soybean region within a bounding box, the segmentation algorithm outputs a binary mask where the soybean pixel value is 1 and other pixel values ​​are 0;

[0091] The binary mask of soybean area is used to separate individual soybeans using image processing algorithms;

[0092] Commonly used image processing algorithms include erosion, dilation, connected domain analysis, etc.

[0093] Corrosion Operation: ;

[0094] in, Represents the pixel after the corrosion operation The new value at

[0095] Represents the displacement in the original image The pixel value at ;

[0096] express Belongs to the structural element S, which defines the neighborhood range and shape of the corrosion operation;

[0097] Expansion operation: ;

[0098] in, Represents the pixel after the dilation operation The new value at

[0099] represents the displacement in the original image ( The pixel value at ;

[0100] express Belongs to the structural element S, which defines the neighborhood range and shape of the dilation operation;

[0101] Connected component analysis divides the mask into independent regions by marking adjacent pixels;

[0102] Example: Erosion, dilation, and connected domain analysis are performed on a soybean region mask to separate overlapping soybeans into multiple independent binary masks;

[0103] Step 400: Analyze each soybean binary mask, calculate the maximum projection of the soybean in the horizontal and vertical directions, obtain the pixel value of the grain length and grain width, convert the pixel value into a real length unit using the camera calibration parameter, and output the grain length and grain width data of all soybean grains in each image frame;

[0104] Specifically:

[0105] The binary mask of each soybean was analyzed;

[0106] Calculate the projection of the mask in the horizontal direction and find the maximum projection width as the pixel value of the grain length;

[0107] Calculate the projection of the mask in the vertical direction and find the maximum projection height as the pixel value of the particle width;

[0108] Use the camera calibration parameters to calculate the conversion coefficient between pixels and actual length;

[0109] Multiply the pixel value by the conversion coefficient to obtain the actual length value (e.g. centimeters) of the soybean grain length and width;

[0110] Example: The mask projection of a soybean grain has a maximum horizontal width of 50 pixels and a maximum vertical height of 30 pixels. The conversion coefficient of the calibration result from pixels to centimeters is 0.02. The length of the soybean grain is 500*0.02=1cm, and the width is 300.02=0.6cm.

[0111] For each image frame, the grain length and width measurement results of all soybeans in it are generated;

[0112] Output the results according to the set format, such as saving as a CSV file;

[0113] Each line of the CSV file corresponds to a soybean in an image frame, including fields such as grain number, frame number, grain length value, and grain width value;

[0114] Example:

[0115] Particle number Frame number Grain length (cm) Grain width (cm) 1 1 1.2 0.7 2 1 0.9 0.6 3 1 1.1 0.8 ... ... ... ... n m 1.0 0.7

[0116] Step 500, setting a soaking swelling grain length threshold, a soaking swelling grain width threshold and a sufficient soaking judgment condition of soybeans, traversing all soybean grain length and grain width data in each image frame in chronological order, selecting an image frame of sufficient soaking after preliminary screening according to the sufficient soaking judgment condition, obtaining all image frames between the image frame of sufficient soaking after preliminary screening and the previous image frame from the soybean soaking video data, and performing pre-processing;

[0117] The soaking swelling grain length threshold of soybean is 1.65 times of the initial soybean grain length, and the soaking swelling grain width threshold of soybean is 1.10 times of the initial soybean. The sufficient soaking judgment condition is that the grain length and grain width data of all soybeans in the image frame are greater than or equal to the soaking swelling grain length threshold and the soaking swelling grain width threshold.

[0118] Step 600, processing all the image frames preprocessed in step 500 according to steps 300 to 400, and obtaining the grain length and grain width measurement results of all soybeans in all image frames between the image frame of the initial screening and the previous image frame, selecting the image frame data of accurate and sufficient soaking after traversing in time order, the image frame data of accurate and sufficient soaking includes the timestamp of the image frame and the grain length and grain width measurement data of all soybeans, and constructing a training data set based on the grain length and grain width measurement results of all soybeans in all image frames generated in step 400 and the image frame data of accurate and sufficient soaking;

[0119] Step 700: construct a sufficient foaming prediction model, and train the sufficient foaming prediction model using a training data set;

[0120] Specifically:

[0121] Based on the training data set, construct the input vector of the model;

[0122] Each sample in the training data set contains multiple soybean grain length and width data;

[0123] We can calculate statistics of these data, such as mean, median, standard deviation, etc., as features;

[0124] For example: average grain length, standard deviation of grain length, median grain width, etc.;

[0125] Each sample corresponds to a specific immersion time point;

[0126] Use soaking time directly as a feature;

[0127] Divide the grain length and width values ​​into several intervals;

[0128] Calculate the distribution frequency of the number of soybean grains in each interval as a feature;

[0129] For example: the proportion of grain length in the range of 0.5-1cm, the frequency of grain width in the range of 1-1.5cm, etc.

[0130] The maximum grain length and minimum grain width of all soybean grains in the training set are used as features;

[0131] Target features:

[0132] According to the desired fully expanded state (such as 1.65 times the length and 1.10 times the width)

[0133] Whether each sample reaches this state is used as a 0 / 1 target label;

[0134] Finally, we can combine the above features into a vector as input to the machine learning model;

[0135] For example:

[0136] Input vector: [mean grain length, standard deviation of grain length, soaking time, frequency of grain length 0.5-1 cm, ..., maximum grain length, target label];

[0137] The full foaming prediction model includes an input layer, a hidden layer and an output layer;

[0138] Let the hidden layer have M neurons;

[0139] For the jth hidden layer neuron, its input is:

[0140]

[0141] in, is the weight from the i-th neuron in the input layer to the j-th neuron in the hidden layer;

[0142] is the input value of the i-th neuron in the input layer;

[0143] is the bias term of the jth hidden layer neuron;

[0144] The output of the hidden layer neurons is:

[0145]

[0146] Where f is the activation function, such as the ReLU function;

[0147] Let the output layer have only one neuron, and the output value is the predicted foaming time;

[0148] The input of the output layer neurons is:

[0149]

[0150] is the weight from the jth neuron in the hidden layer to the neuron in the output layer;

[0151] is the output of the jth neuron in the hidden layer;

[0152] is the output layer bias term;

[0153] During training, the difference between the predicted foaming time and the actual foaming time is used as a loss function (such as mean square error), and the weights and biases are continuously adjusted using an optimization algorithm (such as gradient descent).

[0154] Step 800, setting an acquisition time period, acquiring new soybean soaking image data at the set time interval, processing the new soybean soaking image data according to steps 300 to 400 to generate an input vector, and inputting the input vector into a full soaking prediction model, which outputs predicted full soaking time data;

[0155] The collection time period does not exceed the shortest foaming time in the historical data;

[0156] It should be noted that based on the predicted sufficient soaking time data, image data of soybean soaking is collected at this time point. After processing, the soybean soaking status can be obtained. If it meets the requirements, the beans are directly put into the processing, which can effectively reduce the time of ineffective soaking, thereby improving production efficiency and labor costs. It can realize automated, efficient and high-precision soybean soaking processing, and can adjust the threshold setting according to actual soaking needs, and can accurately predict the time points of different soaking degrees.

[0157] like Figure 2 As shown, an automatic bean placing system is used to execute the automatic bean placing method as described above, comprising:

[0158] Soaking module, used for soaking soybeans and automatically placing beans;

[0159] A data acquisition module, used for acquiring multiple sets of video data of soybean soaking process;

[0160] The data processing module is used to process multiple sets of video data of soybean soaking process and form a training data set, which specifically includes the following steps:

[0161] Step S1, extracting image frames from each group of soybean soaking video data at a set interval time, and preprocessing all the extracted image frames;

[0162] Step S2, using a target detection algorithm to locate the soybean region in the image frame, segmenting the soybean region from the background, obtaining a binary mask of the soybean region, using an image processing algorithm to separate individual soybeans from the binary mask of the soybean region, and generating an independent binary mask for each soybean;

[0163] Step S3, analyzing each soybean binary mask, calculating the maximum projection of the soybean in the horizontal and vertical directions, obtaining the pixel values ​​of the grain length and grain width, converting the pixel values ​​into real length units using the camera calibration parameters, and outputting the grain length and grain width data of all soybean grains in each image frame;

[0164] Step S4, setting a soaking swelling grain length threshold, a soaking swelling grain width threshold and a sufficient soaking judgment condition of soybeans, traversing all soybean grain length and grain width data in each image frame in chronological order, selecting an image frame of sufficient soaking after preliminary screening according to the sufficient soaking judgment condition, obtaining all image frames between the image frame of sufficient soaking after preliminary screening and the previous image frame from the soybean soaking video data, and performing pre-processing;

[0165] Step S5, processing all the image frames preprocessed in step S4 according to steps S2 to S3, and obtaining the grain length and grain width measurement results of all soybeans in all image frames between the image frame of the initial screening and the previous image frame, selecting the image frame data of accurate and sufficient soaking after traversing in chronological order, the image frame data of accurate and sufficient soaking includes the timestamp of the image frame and the grain length and grain width measurement data of all soybeans, and constructing a training data set based on the grain length and grain width measurement results of all soybeans in all image frames generated in step S3 and the image frame data of accurate and sufficient soaking;

[0166] The prediction module is used to construct a sufficient soaking prediction model, and use a training data set to train the sufficient soaking prediction model, set an acquisition time period, collect new soybean soaking image data at the set time interval, process the new soybean soaking image data according to steps S2 to S3 to generate an input vector, and input it into the sufficient soaking prediction model, and the sufficient soaking prediction model outputs predicted sufficient soaking time data.

[0167] The present invention also discloses a storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the steps in the automatic bean placing method as described above can be executed.

[0168] The present embodiment is described above, but the present embodiment is not limited to the above-mentioned specific implementation manner. The above-mentioned specific implementation manner is merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, which are all within the protection of the present embodiment.

Claims

1. An automatic bean placing method, characterized in that: The following steps are involved: Step 100, obtaining multiple sets of soybean soaking video data; Step 200, extracting image frames from each set of soybean soaking video data at a set interval, and preprocessing all the extracted image frames; Step 300: Use the target detection algorithm to locate the soybean region in the image frame, segment the soybean region from the background, obtain a binary mask of the soybean region, separate individual soybeans from the binary mask of the soybean region using an image processing algorithm, and generate an independent binary mask for each soybean; Step 400: Analyze each soybean binary mask, calculate the maximum projection of the soybean in the horizontal and vertical directions, obtain the pixel value of the grain length and grain width, convert the pixel value into a real length unit using the camera calibration parameter, and output the grain length and grain width data of all soybean grains in each image frame; Step 500, setting a soaking swelling grain length threshold, a soaking swelling grain width threshold and a sufficient soaking judgment condition of soybeans, traversing all soybean grain length and grain width data in each image frame in chronological order, selecting an image frame of sufficient soaking after preliminary screening according to the sufficient soaking judgment condition, obtaining all image frames between the image frame of sufficient soaking after preliminary screening and the previous image frame from the soybean soaking video data, and performing pre-processing; Step 600, processing all the image frames preprocessed in step 500 according to steps 300 to 400, and obtaining the grain length and grain width measurement results of all soybeans in all image frames between the image frame of the initial screening and the previous image frame, selecting the image frame data of accurate and sufficient soaking after traversing in time order, the image frame data of accurate and sufficient soaking includes the timestamp of the image frame and the grain length and grain width measurement data of all soybeans, and constructing a training data set based on the grain length and grain width measurement results of all soybeans in all image frames generated in step 400 and the image frame data of accurate and sufficient soaking; Step 700: construct a sufficient foaming prediction model, and train the sufficient foaming prediction model using a training data set; Specifically: Based on the training data set, construct the input vector of the model; Constructing a full foaming prediction model, the full foaming prediction model includes an input layer, a hidden layer and an output layer; Step 800, set an acquisition time period, collect new soybean soaking image data at the set time interval, process the new soybean soaking image data according to steps 300 to 400 to generate an input vector, and input the input vector into a sufficient soaking prediction model, which outputs predicted sufficient soaking time data.

2. The automatic bean placing method according to claim 1, characterized in that: Image frames from each set of soybean soaking video data are extracted at set intervals, and all extracted image frames are preprocessed as follows: Use the video processing library to read the soybean soaking video file; Set the time interval for extracting image frames; Traverse the video and use the read() function to extract the image frame at the set time point; All extracted image frames are preprocessed.

3. The automatic bean placing method according to claim 2, characterized in that: Analyze each soybean binary mask, calculate the maximum projection of the soybean in the horizontal and vertical directions, obtain the pixel value of the grain length and width, use the camera calibration parameters to convert the pixel value into a real length unit, and output the grain length and width data of all soybean grains in each image frame; Specifically: The binary mask of each soybean was analyzed; Calculate the projection of the mask in the horizontal direction and find the maximum projection width as the pixel value of the grain length; Calculate the projection of the mask in the vertical direction and find the maximum projection height as the pixel value of the particle width; Use the camera calibration parameters to calculate the conversion coefficient between pixels and actual length; Multiply the pixel value by the conversion coefficient to obtain the actual length and width of the soybean grain; For each image frame, the grain length and width measurement results of all soybeans in it are generated.

4. The automatic bean placing method according to claim 3, characterized in that: The threshold value of soybean grain length after soaking and swelling is 1.65 times of the initial soybean grain length, and the threshold value of soybean grain width after soaking and swelling is 1.10 times of the initial soybean grain length.

5. The automatic bean placing method according to claim 4, characterized in that: The sufficient swelling judgment condition is that the grain length and grain width data of all soybeans in the image frame are greater than or equal to the soaking swelling grain length threshold and the soaking swelling grain width threshold.

6. The automatic bean placing method according to claim 5, characterized in that: The hidden layer has M neurons; For the jth hidden layer neuron, its input is: Z j =∑(w ij *x i )+b j ; Among them, w ij is the weight from the i-th neuron in the input layer to the j-th neuron in the hidden layer; x i is the input value of the i-th neuron in the input layer; b j is the bias term of the jth hidden layer neuron; The output of the hidden layer neurons is: h j =f(Z j ); Among them, f is the activation function.

7. The automatic bean placing method according to claim 6, characterized in that: The output layer has only one neuron, and the output value is the predicted foaming time; The input of the output layer neurons is: z=∑(v j *h j )+c; v j is the weight from the jth neuron in the hidden layer to the neuron in the output layer; h j is the output of the jth neuron in the hidden layer; c is the output layer bias term.

8. The automatic bean placing method according to claim 7, characterized in that: During training, the difference between the predicted foaming time and the actual foaming time is used as the loss function, and the weights and biases are continuously adjusted using an optimization algorithm.

9. An automatic bean placing system, characterized in that: It is used to execute an automatic bean placing method as claimed in any one of claims 1 to 8, comprising: Soaking module, used for soaking soybeans and automatically placing beans; A data acquisition module, used for acquiring multiple sets of video data of soybean soaking process; The data processing module is used to process multiple sets of video data of soybean soaking process and form a training data set, which specifically includes the following steps: Step S1, extracting image frames from each group of soybean soaking video data at a set interval time, and preprocessing all the extracted image frames; Step S2, using a target detection algorithm to locate the soybean region in the image frame, segmenting the soybean region from the background, obtaining a binary mask of the soybean region, using an image processing algorithm to separate individual soybeans from the binary mask of the soybean region, and generating an independent binary mask for each soybean; Step S3, analyzing each soybean binary mask, calculating the maximum projection of the soybean in the horizontal and vertical directions, obtaining the pixel values ​​of the grain length and grain width, converting the pixel values ​​into real length units using the camera calibration parameters, and outputting the grain length and grain width data of all soybean grains in each image frame; Step S4, setting a soaking swelling grain length threshold, a soaking swelling grain width threshold and a sufficient soaking judgment condition of soybeans, traversing all soybean grain length and grain width data in each image frame in chronological order, selecting an image frame of sufficient soaking after preliminary screening according to the sufficient soaking judgment condition, obtaining all image frames between the image frame of sufficient soaking after preliminary screening and the previous image frame from the soybean soaking video data, and performing pre-processing; Step S5, processing all the image frames preprocessed in step S4 according to steps S2 to S3, and obtaining the grain length and grain width measurement results of all soybeans in all image frames between the image frame of the initial screening and the previous image frame, selecting the image frame data of accurate and sufficient soaking after traversing in chronological order, the image frame data of accurate and sufficient soaking includes the timestamp of the image frame and the grain length and grain width measurement data of all soybeans, and constructing a training data set based on the grain length and grain width measurement results of all soybeans in all image frames generated in step S3 and the image frame data of accurate and sufficient soaking; The prediction module is used to construct a sufficient soaking prediction model, and use a training data set to train the sufficient soaking prediction model, set an acquisition time period, collect new soybean soaking image data at the set time interval, process the new soybean soaking image data according to steps S2 to S3 to generate an input vector, and input it into the sufficient soaking prediction model, and the sufficient soaking prediction model outputs predicted sufficient soaking time data.

10. A storage medium, characterized in that: It stores non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by a computer, the steps in the automatic bean placing method as described in any one of claims 1-8 can be executed.

Images