All-weather apple fruit position sensing method based on multi-modal data

Abstract

The invention discloses an all-weather apple fruit position sensing method based on multi-modal data, and the method comprises the steps: obtaining the RGB image, distance and infrared radiation information of a fruit in daytime and night environments through a (TOF) depth camera, inputting the RGB image, distance and infrared radiation information into an improved five-channel Faster-RCNN model for training, and obtaining a model suitable for different illumination environments; according to the improved Faster-RCNN model, three channels of a data input module are changed into five channels, and common convolution is changed into depth separable convolution, so that the operand is reduced. During detection, image data are simultaneously and respectively input into models obtained by training in daytime and night illumination environments, weight division is performed on results of the two models according to illumination intensity, and specific position information of the apple is comprehensively output. According to the method, the performance of the apple picking robot in a dark light environment is remarkably improved, and the performance is better than that of a traditional RGB detection method in a strong light environment.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection in computer vision, and in particular to an all-weather apple fruit position perception method based on multimodal data. Technical Background

[0002] With the development of society, people's living standards are gradually improving, and the demand for fruits is gradually increasing. As one of the most common fruits, the demand for apples is also increasing. my country is a big agricultural country and a big fruit producer. When apples are ripe, picking them is a big problem.

[0003] Nowadays, labor costs are extremely high, and the efficiency of manual picking is also very low. Some areas have begun to use picking robots for automatic picking, but the current common fruit positioning method based on RGB images has a low accuracy rate, and the detection effect is greatly affected by the light intensity. When the light is poor, the detection effect of the fruit position is extremely poor, so the picking robot cannot work normally at night, which greatly affects the picking efficiency. In addition, when the fruit is partially blocked by branches, the traditional recognition method cannot detect this special situation in time and handle it accordingly. Directly extending the robotic arm for picking is very likely to cause damage to the robot and cause economic losses.

[0004] In recent years, various sensor technologies have developed rapidly. For example, the Kinect series of depth cameras released by Microsoft can not only collect common RGB information, but also obtain corresponding distance information and infrared radiation information. This information is of great significance for the task of sensing the position of apples and judging whether the fruits are blocked. In addition, deep learning technology has also developed rapidly in recent years, and some model optimization methods have emerged. This makes the computing power requirements of some commonly used models lower, and they can be better migrated to mobile terminals for use.

[0005] Therefore, it is an urgent problem for technical personnel in this field to propose an all-weather apple fruit position perception method based on multimodal data to solve the difficulties existing in the prior art. Summary of the invention

[0006] In view of this, the present invention proposes an all-weather apple fruit position perception method based on multimodal data. By integrating RGB information, depth information, and infrared radiation information, the present invention effectively solves the problems encountered by current picking robots, such as weak light, low accuracy, and inability to identify whether the fruit is blocked by branches. The Faster-RCNN model is optimized using the deep separable convolution technology to further reduce its computing power requirements and improve its practicality on mobile platforms.

[0007] The present invention is an all-weather apple fruit position sensing method based on multimodal data, which is characterized by comprising the following steps:

[0008] Step 1: Use the (TOF) depth camera to collect multiple sets of raw data in two lighting environments: daytime and nighttime. Each set includes RGB information, distance information, and infrared radiation information.

[0009] Step 2: Align the collected distance information, infrared radiation information and RGB information; first, adjust the resolution of the RGB image data to a predetermined size, and perform the same resolution conversion on the distance information and infrared radiation information to make their matrix sizes consistent. This step ensures that the three types of information are aligned. Next, normalize the values ​​of the distance information matrix and the infrared radiation information matrix, and limit their value range to between 0 and 255 to keep consistent with the data range of the RGB image; the normalization calculation formula is:

[0010]

[0011] Among them, value is an element in the array, min_value and max_value are the minimum and maximum values ​​of the array respectively, and normalized_value is the normalized value;

[0012] Step 3: Manually label the collected data, use labelimg software to add labels to the data, set two labels: "fruits that can be picked" and "fruits blocked by branches" respectively; export the labeling results to an XML format file supported by Faster-RCNN, and finally form a complete data set; then, divide the data set into training set, test set and validation set in a ratio of 8:1:1;

[0013] Step 4: Improve and optimize the Faster-RCNN model. First, adjust the input channels of the model from the original three channels to five channels. The input of the conventional model is width*height*3 (width*height*number of channels), which is changed to width*height*5 (width*height*number of channels) after optimization. In addition, replace the original ordinary convolution operation with depthwise separable convolution, which can significantly reduce the amount of calculation, thereby reducing the demand for computing power.

[0014] Step 5: Train the data for both daytime and nighttime lighting environments respectively; input the training set and validation set data into the model for training, and use the test set to evaluate the model effect after training; train five times in each environment, select the model with the best test effect and save it, and finally obtain the optimized Faster-RCNN model files suitable for the two environments for subsequent detection operations;

[0015] Step 6: Input the above pre-processed RGB information, distance information and infrared radiation information into the trained optimized Faster-RCNN model; according to the light intensity recorded synchronously when the input data is collected, the model detection results under the two lighting environments of day and night are weighted, and finally the accurate position positioning result of the apple fruit is obtained, which specifically includes the following sub-steps:

[0016] Step 6.1: Input the data to be tested into the model under both daytime and nighttime lighting environments, and obtain the light intensity during data collection; Under normal circumstances, the average light intensity during the day in China is about 8000 lux, while the light intensity at night is about 2000 lux; Lux (symbol: lx) is the legal unit of light intensity;

[0017] Step 6.2: When the light intensity is lower than 2000 lux, the weight of the model detection result trained under daytime light is set to 0, and the weight of the model detection result trained under nighttime light is set to 1;

[0018] Step 6.3: When the light intensity is between 2000lux and 8000lux, the weight calculation formulas of the two models are as follows:

[0019]

[0020] Among them, weightA represents the weight of the detection result of the model trained under daylight, and weightB represents the weight of the detection result of the model trained under night light;

[0021] Step 6.4: When the light intensity reaches 8000 lux or above, the weight of the model detection result trained under night light is set to 0, and the weight of the model detection result trained under daylight is set to 1;

[0022] Step 6.5: The model with a weight greater than 50% is identified as the main detection model and is recorded as model m , and the weight of the detection result is recorded as weight m , the other model is the secondary detection model, denoted as model s , and the weight of the detection result is recorded as weight s ;

[0023] Step 6.6: In the Faster R-CNN model, the bounding box and confidence of the target are determined by the joint calculation of the region proposal network (RPN) and the classifier; several candidate regions are generated by the region proposal network, which are generated by combining anchor boxes with the feature map of the input image at different positions; each anchor box has a fixed size and ratio and covers different areas of the image; the region proposal network predicts two types of information for each anchor box: one is the probability that the anchor box belongs to the foreground (target) or background (non-target), and the other is the bounding box regression parameters used to accurately locate the target to adjust the anchor box to fit the target edge more accurately. These candidate regions are processed by non-maximum suppression (NMS) to remove highly overlapping redundant boxes and only retain bounding boxes with high correlation with the predicted target. These screened candidate regions are sent to the subsequent classifier for further analysis. The classifier calculates the target category and its corresponding confidence of each region, and the bounding box regressor is used to further optimize the precise position and size of the bounding box; finally, the model outputs the classification confidence and target bounding box coordinates of the target category of each candidate region;

[0024] Step 6.7: When the overlap of the bounding box areas of the targets detected by the two models reaches 90% or more, they are considered to have detected the same target. The calculation method of the area overlap is as follows: First, find the area of ​​the overlapping part of the target bounding box detected by the two models, multiply it by 2, and then divide it by the model m and model s The sum of the areas of the bounding boxes of the detected targets; the calculation formula for area overlap is as follows:

[0025]

[0026] Where R is the regional overlap, A1 is the model m The area of ​​the detected target bounding box, A2 is the model s The area of ​​the detected object bounding box, A intersection is the area of ​​the overlapped part of the two bounding boxes;

[0027] Step 6.8: During the detection process, when the model m A target is detected, and the model s When the target is not detected, the confidence of the target is taken from the model m The confidence of the target is obtained, and the coordinate information of the bounding box is obtained using the model m The coordinate information of the target is obtained;

[0028] Step 6.9: During the detection process, when the models An object is detected, but the model m When not detected, the confidence of the target is directly set to 0;

[0029] Step 6.10: During the detection process, when the model m and model s When the same target is detected and the detection labels are consistent, the confidence of the target is model m The confidence of the target is multiplied by weight m , plus model s The confidence of the target is multiplied by weight s , the coordinate information of the bounding box uses the model m The coordinate information of the target is obtained; the confidence calculation formula of the target is as follows:

[0030] C=weight m ×C m +weight s ×C s ,

[0031] Where C is the confidence of the target, C m For model m The confidence of the target obtained, C s For model s The confidence level of the target obtained;

[0032] Step 6.11: During the detection process, when the model m and model s When the detection labels of the same target are different, the confidence of the target is directly set to 0;

[0033] Step 6.12: Set the threshold of the final detection confidence to 0.85. When the confidence is higher than 0.85, the detection is considered successful, and the coordinate information of the target and the corresponding category label are returned; BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the present disclosure is further described below with reference to the accompanying drawings and implementation cases.

[0035] Figure 1 It is a schematic diagram of the implementation steps of the all-weather apple fruit position sensing method based on multimodal data;

[0036] Figure 2 It is a picture instance of collected and processed RGB information, distance information, and infrared radiation information;

[0037] Figure 3This is a schematic diagram of the final test results under the condition of a light intensity of 9000 lux;

[0038] Figure 4 This is a schematic diagram of the final test results under the condition of a light intensity of 5000 lux;

[0039] Figure 5 This is a schematic diagram of the final test results under the condition of a light intensity of 1500 lux; DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] The present invention is an all-weather apple fruit position sensing method based on multimodal data, and a specific embodiment thereof is characterized in that it comprises the following steps:

[0042] Step 1: The data for this implementation case comes from a public dataset called KFujiRGB-DSdataset on Kaggle; this dataset collects data under both daytime and nighttime lighting conditions through a (TOF) depth camera, including aligned RGB information, distance information, infrared radiation information, and ambient light intensity during data collection; RGB information, distance information, and infrared radiation information are all provided in JPEG image format with a resolution of 548×353, and the light intensity is expressed in decimal integer format; in this implementation case, 500 sets of data for daytime lighting environments and 500 sets of data for nighttime lighting environments were selected from the dataset for implementation case analysis;

[0043] Step 2: The data set used in this example has completed the data alignment operation; specifically, the (TOF) depth camera manufacturer will provide the camera intrinsic and extrinsic parameters required for the alignment operation and the specific data alignment operation process; the data set used in this example has not been normalized, and then the values ​​of the distance information matrix and the infrared radiation information matrix are normalized to limit their value range to between 0 and 255, consistent with the RGB image; the normalization calculation formula is:

[0044]

[0045] Among them, value is an element in the array, min_value and max_value are the minimum and maximum values ​​of the array respectively, and normalized_value is the normalized value;

[0046] Step 3: Use labelimg annotation software to label the RGB images, set two labels "A" and "B", label A represents "fruits that can be picked", label B represents "fruits blocked by branches", and complete the annotation of RGB images in 1000 sets of data according to the definition of labels, and save the annotation information as XML format files; divide them into training set, test set, and validation set in a ratio of 8:1:1; the specific number is 800 groups, 100 groups, and 100 groups respectively, with daytime and nighttime lighting environments accounting for half each;

[0047] Step 4: Improve and optimize the Faster-RCNN model. First, adjust the input channels of the model from the original three channels to five channels. The input of the conventional model is 548x353x3 (width x height x number of channels), which is changed to 548x353x5 (width x height x number of channels) after optimization. In addition, replace the original ordinary convolution operation with depthwise separable convolution, which can significantly reduce the amount of calculation, thereby reducing the demand for computing power.

[0048] Step 5: Train the data for both daytime and nighttime lighting environments respectively; input the training set and validation set data into the model for training, and use the test set to evaluate the model effect after training; train five times in each environment, select the model with the best test effect and save it, and finally obtain the optimized Faster-RCNN model files suitable for the two environments for subsequent detection operations;

[0049] Step 6: Input the above pre-processed RGB information, distance information and infrared radiation information into the trained optimized Faster-RCNN model; according to the light intensity recorded synchronously when the input data is collected, the model detection results under the two lighting environments of day and night are weighted, and finally the accurate position positioning result of the apple fruit is obtained, which specifically includes the following sub-steps:

[0050] Step 6.1: Input the data to be detected into the model under both daytime and nighttime lighting environments at the same time, and obtain the light intensity when the data is collected; Under normal circumstances, the average light intensity during the day in China is about 8000 lux, while the light intensity at night is about 2000 lux; Lux (symbol: lx) is the legal unit of light intensity; This implementation case selects three sets of data from the KFujiRGB-DSdataset public data set, each set of data contains RGB information, distance information, infrared radiation information collected and aligned by a (TOF) depth camera, and the ambient light intensity at the time of data collection; This implementation case selects three sets of images with light intensities of 9000 lux (daytime), 5000 lux (evening), and 1500 lux (nighttime) for testing to demonstrate the specific implementation process of the present invention; The images to be detected selected at the three light intensities and their corresponding detection effect diagrams are shown in the attached figure Figure 3 , Figure 4 , Figure 5 As shown;

[0051] Step 6.2: When the light intensity is lower than 2000 lux, the weight of the model detection result trained under daytime light is set to 0, and the weight of the model detection result trained under nighttime light is set to 1;

[0052] Step 6.3: When the light intensity is between 2000lux and 8000lux, the weight calculation formulas of the two models are as follows:

[0053]

[0054] Among them, weightA represents the weight of the detection result of the model trained under daylight, and weightB represents the weight of the detection result of the model trained under night light;

[0055] Step 6.4: When the light intensity reaches 8000 lux or above, the weight of the model detection result trained under night light is set to 0, and the weight of the model detection result trained under daylight is set to 1;

[0056] Step 6.5: The model with a weight greater than 50% is identified as the main detection model and is recorded as model m , and the weight of the detection result is recorded as weight m , the other model is the secondary detection model, denoted as model s , and the weight of the detection result is recorded as weight s ;

[0057] Step 6.6: When the light intensity of the image to be detected is 1500 lux, it is determined that the light intensity of the image to be detected meets the calculation method of two weights when the light intensity is lower than 2000 lux. Therefore, the weight of the detection result of the model trained under daylight is set to 0, and the weight of the detection result of the model trained under nightlight is set to 1. That is, the model trained under nightlight is identified as the main detection model, denoted as model m , and the weight of the detection result is recorded as weight m , weight m The value of is 1; the model trained under daylight is identified as the secondary detection model, denoted as model s , and the weight of the detection result is recorded as weight s ,weight s The value of is 0;

[0058] Step 6.7: When the light intensity corresponding to the image to be detected is 5000 lux, it is determined that the light intensity corresponding to the image to be detected meets the calculation method of two weights when the light intensity is between 2000lux-8000lux, so it is calculated according to the formula:

[0059]

[0060] That is, the model trained under night illumination is identified as the main detection model, denoted as model m , and the weight of the detection result is recorded as weight m , weight m The value of is 0.625, and the model trained under daylight is identified as the secondary detection model, denoted as model s , and the weight of the detection result is recorded as weight s , weight s The value of is 0.375;

[0061] Step 6.8: When the light intensity of the image to be detected is 9000 lux, it is determined that the light intensity of the image to be detected meets the calculation method of two weights when the light intensity is higher than 8000 lux. Therefore, the weight of the detection result of the model trained under night light is set to 0, and the weight of the detection result of the model trained under day light is set to 1. That is, the model trained under day light is identified as the main detection model, recorded as model m , and the weight of the detection result is recorded as weight m , weight m The value of is 1; the model trained under night illumination is identified as the secondary detection model, denoted as model s , and the weight of the detection result is recorded as weight s ,weights The value of is 0;

[0062] Step 6.9: For other light intensity conditions other than those in the embodiment, weight calculation should be performed according to the specific steps in the claims to ensure that accurate detection effects can be achieved under various light conditions;

[0063] Step 6.10: In the Faster R-CNN model, the bounding box and confidence of the target are determined by the joint calculation of the region proposal network (RPN) and the classifier; several candidate regions are generated by the region proposal network, which are generated by combining anchor boxes with the feature map of the input image at different positions; each anchor box has a fixed size and ratio and covers different areas of the image; the region proposal network predicts two types of information for each anchor box: one is the probability that the anchor box belongs to the foreground (target) or background (non-target), and the other is the bounding box regression parameters used to accurately locate the target to adjust the anchor box to fit the target edge more accurately. These candidate regions are processed by non-maximum suppression (NMS) to remove highly overlapping redundant boxes and only retain bounding boxes with high correlation with the predicted target. These screened candidate regions are sent to the subsequent classifier for further analysis. The classifier calculates the target category and its corresponding confidence of each region, and the bounding box regressor is used to further optimize the precise position and size of the bounding box; finally, the model outputs the classification confidence and target bounding box coordinates of the target category of each candidate region;

[0064] Step 6.11: When the overlap of the bounding box areas detected by the two models reaches 90% or more, they are considered to have detected the same target. The calculation method of the area overlap is as follows: first calculate the area of ​​the overlapping part of the bounding box detected by the two models, multiply it by 2, and then divide it by the model m and model s The sum of the areas of the bounding boxes of the detected targets; the calculation formula for area overlap is as follows:

[0065]

[0066] Where R is the regional overlap, A1 is the model m The area of ​​the detected target bounding box, A2 is the model s The area of ​​the detected object bounding box, A intersection is the area of ​​the overlapped part of the two bounding boxes;

[0067] Step 6.12: During the detection process, when the model m A target is detected, and the model sWhen the target is not detected, the confidence of the target is taken from the model m The confidence of the target is obtained, and the coordinate information of the bounding box is obtained using the model m The coordinate information of the target is obtained;

[0068] Step 6.13: During the detection process, when the model s An object is detected, but the model m When not detected, the confidence of the target is directly set to 0;

[0069] Step 6.14: During the detection process, when the model m and model s When the same target is detected and the detection labels are consistent, the confidence of the target is model m The confidence of the target is multiplied by weight m , plus model s The confidence of the target is multiplied by weight s , the coordinate information of the bounding box uses the model m The coordinate information of the target is obtained; this implementation case selects a model m and model s Detecting targets with the same label as implementation cases to demonstrate the specific implementation process, model m The confidence level of the target is 0.96, and the model s The confidence level of the target is 0.93. The confidence level calculation formula of the target is as follows:

[0070] C=weight m ×C m +weight s ×C s ,

[0071] Where C is the confidence of the target, C m For model m The confidence of the target obtained, C s For model s The confidence level of the target obtained;

[0072] Step 6.15: When the light intensity is 1500 lux, the model m For the model trained under night illumination, weight m The value of model is 1; s For the model trained under daylight, weight s The value of is 0; calculated according to the formula:

[0073] C = 1 × 0.96 + 0 × 0.93

[0074] Through calculation, the final confidence level of the target is 0.96;

[0075] Step 6.16: When the light intensity is 5000 lux, the model m For the model trained under night illumination, weight m The value of model is 0.625; s For the model trained under daylight, weight s The value of is 0.375; calculated according to the formula:

[0076] C = 0.625 × 0.96 + 0.375 × 0.93

[0077] Through calculation, it can be obtained that the final confidence level of the target is 0.94;

[0078] Step 6.17: When the light intensity is 9000 lux, the model m For the model trained under daylight, weight m The value of model is 1; s For the model trained under night illumination, weight s The value of is 0; calculated according to the formula:

[0079] C = 1 × 0.96 + 0 × 0.93

[0080] Through calculation, the final confidence level of the target is 0.96;

[0081] Step 6.18: During the detection process, when the model m and model s When the detection labels of the same target are different, the confidence of the target is directly set to 0;

[0082] Step 6.19: Set the threshold of the final detection confidence to 0.85. When the confidence is higher than 0.85, the detection is considered successful, and the coordinate information of the target and the corresponding category label are returned.

[0083] Finally, the information returned by the method of the present invention may be helpful for apple picking robots to locate the position of harvestable fruits.

[0084] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the scope of the specific implementation methods. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for sensing the position of apple fruit in all weather conditions based on multimodal data, characterized in that The following steps are involved: Step 1: Use the (TOF) depth camera to collect multiple sets of raw data in two lighting environments: daytime and nighttime. Each set includes RGB information, distance information, and infrared radiation information. Step 2: Align the collected distance information, infrared radiation information and RGB information; first, adjust the resolution of the RGB image data to a predetermined size, and perform the same resolution conversion on the distance information and infrared radiation information to make their matrix sizes consistent. This step ensures that the three types of information are aligned. Next, normalize the values ​​of the distance information matrix and the infrared radiation information matrix, and limit their value range to between 0 and 255, so as to be consistent with the data range of the RGB image; The normalized calculation formula is: Among them, value is an element in the array, min_value and max_value are the minimum and maximum values ​​of the array respectively, and normalized_value is the normalized value; Step 3: Manually annotate the collected data, use data annotation software to add annotations to the data, set two labels: "fruits that can be picked" and "fruits blocked by branches" respectively; export the annotation results to an XML format file supported by Faster-RCNN, and finally form a complete data set; then, divide the data set into training set, test set and validation set in a ratio of 8:1:1; Step 4: Improve and optimize the Faster-RCNN model. First, adjust the input channels of the model from the original three channels to five channels. The input of the conventional model is width*height*3 (width*height*number of channels), which is changed to width*height*5 (width*height*number of channels) after optimization. In addition, replace the original ordinary convolution operation with depthwise separable convolution, which can significantly reduce the amount of calculation, thereby reducing the demand for computing power. Step 5: Train the data for both daytime and nighttime lighting environments respectively; input the training set and validation set data into the model for training, and use the test set to evaluate the model effect after training; train five times in each environment, select the model with the best test effect and save it, and finally obtain the optimized Faster-RCNN model files suitable for the two environments for subsequent detection operations; Step 6: Input the above pre-processed RGB information, distance information and infrared radiation information into the trained optimized Faster-RCNN model; according to the light intensity recorded synchronously when the input data is collected, the model detection results under the two lighting environments of day and night are weighted, and finally the accurate position positioning result of the apple fruit is obtained, which specifically includes the following sub-steps: Step 6.1: Input the data to be tested into the model under both daytime and nighttime lighting environments, and obtain the light intensity during data collection; Under normal circumstances, the average light intensity during the day in China is about 8000 lux, while the light intensity at night is about 2000 lux; Lux (symbol: lx) is the legal unit of light intensity; Step 6.2: When the light intensity is lower than 2000 lux, the weight of the model detection result trained under daytime light is set to 0, and the weight of the model detection result trained under nighttime light is set to 1; When the light intensity is between 2000lux and 8000lux, the weight calculation formulas of the two models are as follows: Among them, weightA represents the weight of the detection result of the model trained under daylight, and weightB represents the weight of the detection result of the model trained under nightlight. When the light intensity reaches 8000 lux or above, the weight of the detection result of the model trained under nightlight is set to 0, and the weight of the detection result of the model trained under daylight is set to 1. The model with a weight greater than 50% is identified as the main detection model, denoted as model m , and the weight of the detection result is recorded as weight m , the other model is the secondary detection model, denoted as model s , and the weight of the detection result is recorded as weight s ; Step 6.3: In the Faster R-CNN model, the bounding box and confidence of the target are determined by the joint calculation of the region proposal network (RPN) and the classifier; several candidate regions are generated by the region proposal network, which are generated by combining anchor boxes with the feature map of the input image at different positions; each anchor box has a fixed size and ratio and covers different areas of the image; the region proposal network predicts two types of information for each anchor box: one is the probability that the anchor box belongs to the foreground (target) or background (non-target), and the other is the bounding box regression parameters used to accurately locate the target to adjust the anchor box to fit the target edge more accurately. These candidate regions are processed by non-maximum suppression (NMS) to remove highly overlapping redundant boxes and only retain bounding boxes with high correlation with the predicted target. These screened candidate regions are sent to the subsequent classifier for further analysis. The classifier calculates the target category and its corresponding confidence of each region, and the bounding box regressor is used to further optimize the precise position and size of the bounding box; finally, the model outputs the classification confidence and target bounding box coordinates of the target category of each candidate region; Step 6.4: When the overlap of the bounding box areas of the targets detected by the two models reaches 90% or more, they are considered to have detected the same target. The calculation method of the area overlap is as follows: first calculate the area of ​​the overlapping part of the target bounding box detected by the two models, multiply it by 2, and then divide it by the model m and model s The sum of the areas of the bounding boxes of the detected targets; the calculation formula for area overlap is as follows: Where R is the regional overlap, A1 is the model m The area of ​​the detected target bounding box, A2 is the model s The area of ​​the detected object bounding box, A intersection is the area of ​​the overlapped part of the two bounding boxes; Step 6.5: During the detection process, when the model m A target is detected, and the model s When the target is not detected, the confidence of the target is taken from the model m The confidence of the target is obtained, and the coordinate information of the bounding box is obtained using the model m The coordinate information of the target is obtained; Step 6.6: During the detection process, when the model s An object is detected, but the model m When not detected, the confidence of the target is directly set to 0; Step 6.7: During the detection process, when the model m and model s When the same target is detected and the detection labels are consistent, the confidence of the target is model m The confidence of the target is multiplied by weight m , plus model s The confidence of the target is multiplied by weight s , the coordinate information of the bounding box uses the model m The coordinate information of the target is obtained; the confidence calculation formula of the target is as follows: C=weight m ×C m +weight s ×C s , Where C is the confidence of the target, C m For model m The confidence of the target obtained, C s For model s The confidence level of the target obtained; Step 6.8: During the detection process, when the model m and model s When the detection labels of the same target are different, the confidence of the target is directly set to 0; Step 6.9: Set the threshold of the final detection confidence to 0.

85. When the confidence is higher than 0.85, the detection is considered successful, and the coordinate information of the target and the corresponding category label are returned.

Images