Branch recognition method, device, computer equipment, storage medium

Abstract

The present application relates to a branch recognition method, apparatus, computer device, storage medium, and computer program product. The method includes: obtaining an image to be processed; obtaining key points of a target object according to the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object; based on the key points, simulating the growth direction of the branches of the target object to obtain a simulation result, and verifying the simulation result to obtain a valid simulation result; and obtaining the pose of the simulation result of the target object according to the valid simulation result. Using this method can improve the accuracy of branch recognition.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly to a method and device for branch recognition, a computer device, a storage medium, and a computer program product. Background Art

[0002] With the rapid development of computer vision technology, computer vision technology has been widely used in agricultural automation, greatly reducing the expenditure of labor costs.

[0003] In related technologies, many automated harvesting solutions use visual recognition to accurately locate fruits and vegetables to be harvested, but they cannot accurately identify the positions of branches, which brings certain difficulties to the realization of automated fruit and vegetable picking. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method and device for branch recognition, a computer device, a computer-readable storage medium, and a computer program product that improve the accuracy of plant branch recognition.

[0005] In a first aspect, this application provides a method for branch recognition, which includes:

[0006] Obtain an image to be processed;

[0007] Obtain key points of a target object from the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object;

[0008] Based on the key points, simulate the growth direction of the branches of the target object to obtain a simulation result, and verify the simulation result to obtain a valid simulation result;

[0009] According to the valid simulation result, obtain the pose of the simulation result of the target object.

[0010] In one embodiment, the obtaining key points of the target object from the image to be processed includes:

[0011] Detect the image to be processed to obtain the main trunk information of the target object;

[0012] Identify the key points from the main trunk information of the target object.

[0013] In one embodiment, the simulating the growth direction of the branches of the target object based on the key points to obtain a simulation result and verifying the simulation result to obtain a valid simulation result includes:

[0014] Based on the key points, obtain the depth image of the target object;

[0015] In the depth image of the target object, simulate the growth direction of the branches of the target object to obtain a simulation result;

[0016] Verify the simulation result to obtain a valid simulation result.

[0017] In one embodiment, obtaining the depth image of the target object based on the key points includes:

[0018] Obtain the depth image corresponding to the image to be processed, and obtain the depth information of the key points according to the depth image corresponding to the image to be processed;

[0019] Based on the depth information of the key points, and filter out the noise in the depth image corresponding to the image to be processed according to the set depth threshold to obtain the depth image of the target object.

[0020] In one embodiment, simulating the growth direction of the branches of the target object in the depth image of the target object to obtain a simulation result includes at least one of the following:

[0021] In the depth image of the target object, sample based on the growth line of the target object at a preset first angle to obtain a simulation result; the growth line is determined according to the slope and intercept corresponding to the target object; or

[0022] In the depth image of the target object, starting from the key point, sample with a preset number of line segments and a preset second angle to obtain a simulation result.

[0023] In one embodiment, verifying the simulation result to obtain a valid simulation result includes at least one of the following:

[0024] Calculate the depth missing rate of each simulation result;

[0025] Remove the simulation results with a depth missing rate exceeding the preset first threshold to obtain a valid simulation result; or

[0026] Project the depth image of the target object to obtain the depth image of the projected target object;

[0027] In the depth image of the projected target object, calculate the inefficiency of each simulation result;

[0028] Remove the simulation results with an inefficiency exceeding the preset second threshold to obtain a valid simulation result.

[0029] In one embodiment, after obtaining the valid simulation result, it further includes:

[0030] Compare each valid simulation result to obtain the maturity state of each valid simulation result.

[0031] In one embodiment, obtaining the pose of the simulation result of the target object based on the valid simulation result includes:

[0032] Filtering out the target simulation result from the valid simulation results;

[0033] Calculating the pose based on the target simulation result.

[0034] In one embodiment, filtering out the target simulation result from the valid simulation results includes:

[0035] Calculating the angles between the simulation result and the camera plane, and between the simulation result and the ground respectively;

[0036] Filtering based on the angle between the simulation result and the camera plane and the angle between the simulation result and the ground to obtain the target simulation result.

[0037] In a second aspect, the present application further provides a branch recognition device, which includes:

[0038] An acquisition module, configured to acquire an image to be processed;

[0039] A key point acquisition module, configured to obtain the key points of the target object according to the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object;

[0040] A simulation module, configured to simulate the growth direction of the branches of the target object based on the key points to obtain a simulation result, and verify the simulation result to obtain a valid simulation result.

[0041] A calculation module, configured to obtain the pose of the simulation result of the target object based on the valid simulation result.

[0042] In a third aspect, the present application further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method in any one of the above embodiments.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the method in any one of the above embodiments.

[0044] In a fifth aspect, the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the method in any one of the above embodiments.

[0045] For the above-mentioned branch recognition method, device, computer equipment, storage medium and computer program product, the controller first obtains the image to be processed; obtains the key points of the target object according to the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object; based on the key points, simulates the growth direction of the branches of the target object to obtain a simulation result, and verifies the simulation result to obtain an effective simulation result; according to the effective simulation result, obtains the pose of the simulation result of the target object. Through the above steps, the specific position of the branches of the target object can be accurately located, and the accuracy of branch recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is an application environment diagram of the branch recognition method in an embodiment;

[0047] Figure 2 It is a schematic flowchart of the branch recognition method in an embodiment;

[0048] Figure 3 It is a schematic diagram of key points in an embodiment;

[0049] Figure 4 It is a schematic diagram of the detection of the main trunk information of the target object in an embodiment;

[0050] Figure 5 It is a depth map of the target object in an embodiment;

[0051] Figure 6 It is a schematic diagram of the slope and intercept of the target object in an embodiment;

[0052] Figure 7 It is a schematic diagram of the simulation result in an embodiment;

[0053] Figure 8 It is a schematic diagram of the simulation result after filtering out the missing depth values in an embodiment;

[0054] Figure 9 It is a schematic diagram of the depth image projection in an embodiment;

[0055] Figure 10 It is a structural block diagram of the branch recognition device in an embodiment;

[0056] Figure 11 It is an internal structure diagram of the computer equipment in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The branch recognition method provided by the embodiments of this application can be applied to an application environment as shown in Figure 1 the following. Among them, the terminal 102 communicates with the controller 104 through a network. Among them, the terminal 102 is a harvesting robot. The data storage system can store the data that the controller 104 needs to process, and the data storage system is integrated on the controller 104. The harvesting robot collects the image to be processed and uploads the image to be processed to the controller 104. After obtaining the image to be processed, the controller 104 obtains the key points of the target object from the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object; based on the key points, the growth direction of the branches of the target object is simulated to obtain a simulation result, and the simulation result is verified to obtain an effective simulation result; finally, according to the effective simulation result, the pose of the simulation result of the target object is obtained. It can be understood that this branch recognition method can also be independently applied to the harvesting robot. The harvesting robot collects the image to be processed, and then obtains the key points of the target object from the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object; based on the key points, the growth direction of the branches of the target object is simulated to obtain a simulation result, and the simulation result is verified to obtain an effective simulation result; finally, according to the effective simulation result, the pose of the simulation result of the target object is obtained.

[0059] The process of the controller 104 for image processing can also be implemented by the terminal 102.

[0060] In one embodiment, as shown in Figure 2 the following, a branch recognition method is provided. Taking the method applied to the controller 104 in Figure 1 as an example, the method includes the following steps:

[0061] S202, Obtain the image to be processed.

[0062] Among them, the image to be processed refers to the image that needs to be recognized for branches. For example, the image to be processed is an RGBD image. An RGBD image refers to an RGB image (color image) and a Depth Map (depth image). The RGB image and the depth image are registered, so there is a one-to-one correspondence between the pixel points of the two. The terminal can capture through a camera to obtain the image to be processed containing the target object and upload the image to be processed to the controller.

[0063] S204, Obtain the key points of the target object according to the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object.

[0064] Among them, the target object refers to the object to be recognized in the image to be processed. Exemplarily, if what needs to be recognized is a tomato plant, then the target object is the tomato plant. Among them, the key point represents the intersection point of the branches of the target object and the main trunk, that is, the bottom growth point of each branch in the object to be processed. Exemplarily, combined with Figure 3 as shown, Figure 3 is a schematic diagram of key points in an embodiment. The position of the small hollow black circle in the figure is a key point of the target object.

[0065] Optionally, the controller can first detect the image to be processed to obtain the main trunk information of the target object, and then identify the key points from the main trunk information. This is because the key points all grow on the main trunk of the target object. Therefore, based on the obtained main trunk information, the key points of the target object can be identified faster and more accurately.

[0066] S206, based on the key points, simulate the growth direction of the branches of the target object to obtain a simulation result, and verify the simulation result to obtain a valid simulation result.

[0067] Among them, the simulation result refers to the branches simulated by the controller according to the growth direction of the branches of the target object, and the valid simulation result refers to the simulation result after the validity verification of the simulation result.

[0068] Optionally, the controller can combine the key points with the depth image of the image to be processed to obtain the depth image of the target object, and then in the depth image of the target object, simulate the growth direction of the branches of the target object to obtain a simulation result.

[0069] Optionally, the controller can simulate based on the growth line of the target object to obtain a simulation result. Specifically, the controller samples based on the growth line of the target object at a preset first angle to obtain a simulation result.

[0070] Optionally, the controller can simulate starting from the key points in the depth map of the target object to obtain a simulation result. Specifically, the controller starts from the key points and samples with a preset number of line segments and a preset second angle to obtain a simulation result.

[0071] Specifically, after simulating the growth direction of the branches of the target object to obtain a simulation result, the controller also needs to verify the simulation result to ensure the validity of the simulation result. Generally speaking, the simulation result obtained by the controller before is a rough simulation result. Therefore, it is necessary to further verify the simulation result to obtain a valid simulation result; and because there are plant leaves or messy small branches in the image, these occlusions will affect the accuracy of calculating the pose of the simulation result in the future. Therefore, this part needs to be filtered out.

[0072] Optionally, the simulation results can be verified by the depth values corresponding to the simulation results. For example, calculate the depth missing rate of each simulation result, and obtain valid simulation results based on the depth missing rate.

[0073] S208. Obtain the pose of the simulation result of the target object according to the valid simulation results.

[0074] Specifically, after the controller obtains the valid simulation results, it calculates the pose of the valid simulation results, where the pose refers to the position and orientation of the simulation results, and then obtains the pose of the simulation result of the target object.

[0075] Optionally, after the controller obtains the pose of the simulation result of the target object, it sends the pose of the simulation result to the terminal, and the terminal picks according to the pose of the simulation result. Among them, the pose of the simulation result is also the pose corresponding to the branch of the target object that is actually identified after the controller processes the image to be processed.

[0076] In the above branch recognition method, the controller first obtains the image to be processed; obtains the key points of the target object according to the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object; based on the key points, simulates the growth direction of the branches of the target object to obtain simulation results, and verifies the simulation results to obtain valid simulation results; obtains the pose of the simulation result of the target object according to the valid simulation results. Through the above steps, the specific position of the branches of the target object can be accurately located, and the accuracy of branch recognition is improved.

[0077] In one embodiment, the above recognition of the image to be processed to obtain the key points of the target object in the image to be processed includes: detecting the image to be processed to obtain the main trunk information of the target object; recognizing the main trunk information of the target object to obtain the key points.

[0078] Among them, the main trunk information refers to the information of the main trunk of the target object obtained by detecting the image to be processed, which can be the position information of the main trunk, such as coordinates, or an image including the main trunk of the target object cut out from the image to be processed.

[0079] Optionally, the main trunk information can be obtained by detecting the object to be processed to obtain a target detection frame, and the main trunk information is located in the target detection frame. Therefore, the main trunk information of the target object can be obtained according to the target detection frame.

[0080] The controller first detects the image to be processed to obtain a target detection box, where the target detection box refers to the detection box where the main trunk of the target object is located, and the information of the main trunk of the target object is inside the target detection box. Optionally, the target detection box can be obtained through a pre-trained detection model, specifically combined with Figure 4 as shown in Figure 4 Figure 4 is a schematic diagram of detecting the information of the main trunk of the target object in an embodiment. The black box in the figure is the target detection box, and the content in the target detection box is the recognized information of the main trunk of the target object.

[0081] After obtaining the information of the main trunk of the target object, further identify the information of the main trunk to obtain the key points on the main trunk, that is, detect the content in the target detection box. Optionally, the information of the main trunk can be input into a pre-trained neural network for key point detection to identify the key points.

[0082] Optionally, the information of the main trunk can be obtained by cutting according to the target detection box on the image to be processed, and then the information of the main trunk is detected to obtain the key points.

[0083] In the above embodiment, the controller first detects the image to be processed to obtain the information of the main trunk of the target object, so that after obtaining the information of the main trunk, the key points on the main trunk can be identified faster and more accurately to accurately obtain the key points.

[0084] In one embodiment, based on the key points, simulate the growth direction of the branches of the target object to obtain a simulation result, and verify the simulation result to obtain an effective simulation result, including: obtaining the depth image of the target object based on the key points; in the depth image of the target object, simulating the growth direction of the branches of the target object to obtain a simulation result; verifying the simulation result to obtain an effective simulation result.

[0085] Specifically, after obtaining the key points, obtain the depth image of the target object according to the key points. Optionally, after obtaining the depth information of the key points from the depth image in the image to be processed, for example, the depth value, the depth image of the target object can be obtained.

[0086] Optionally, based on the depth information corresponding to the key points, the depth value of the main trunk corresponding to the key points can be obtained. Since the key points grow on the main trunk, the depth value of the main trunk can be obtained according to the depth value corresponding to the key points, and then the depth image of the target object can be obtained.

[0087] Specifically, in the depth image of the target object, simulate the growth direction of the branches of the target object to obtain a simulation result, and verify the simulation result to obtain an effective simulation result.

[0088] In the above embodiments, first, a depth image of the target object is obtained through key points, and then simulation and verification are performed in the depth map of the target object because depth information is required in both the simulation process and the verification process; since the depth image includes the actual distance between the sensor and the target object, the simulation results can be better simulated in the depth image; secondly, the depth information can also be used to better verify the effectiveness of the simulation results.

[0089] In one embodiment, obtaining the depth image of the target object based on the key points includes: obtaining the depth image corresponding to the image to be processed, and the depth information of the key points obtained according to the depth image corresponding to the image to be processed; based on the depth information of the key points, and filtering the noise in the depth image corresponding to the image to be processed according to the set depth threshold to obtain the depth image of the target object.

[0090] First, obtain the depth image corresponding to the image to be processed. Since each pixel in the depth image corresponds one-to-one with each pixel in the color image, after obtaining the key points, the depth information of the key points can be obtained from the depth image corresponding to the image to be processed.

[0091] Specifically, first obtain the depth of the main trunk through the depth information of the key points, and then remove the noise in the depth image according to the preset depth threshold to obtain the complete depth mask of the target object, that is, the depth image of the target object. This is because the key points are located on the main trunk of the target object, so the depth of the main trunk can be obtained according to the depth information of the key points.

[0092] Among them, the set depth threshold is set according to the type of the target object. Exemplarily, if the target object is a tomato plant and the corresponding depth range of the plant is 10 - 12, then the preset depth threshold can be set according to the depth range of the plant to remove the noise. In this way, on the basis of obtaining the depth image of the main trunk, the noise can be filtered to obtain the complete depth image of the target object. It should be noted that the depth range of 10 - 12 corresponding to the tomato plant in this embodiment only serves as an explanation and does not represent the actual depth range of the tomato plant being 10 - 12.

[0093] Optionally, select a key point according to a preset rule, such as the bottommost key point, and then obtain the depth of the main trunk according to the depth information of the bottommost key point.

[0094] Figure 5 For the depth map of the target object in one embodiment, in combination with Figure 5 and Figure 4 as shown, Figure 5 the objects other than the target object in Figure 4 have been removed, that is,Figure 4 Remove the cabinet in

[0095] In one embodiment, in the depth image of the target object, simulating the growth direction of the branches of the target object to obtain a simulation result includes at least one of the following: in the depth image of the target object, sampling based on the growth line of the target object at a preset first angle to obtain a simulation result; the growth line is determined according to the slope and intercept corresponding to the target object; or in the depth image of the target object, starting from a key point, sampling with a preset number of line segments and a preset second angle to obtain a simulation result.

[0096] Optionally, in the depth image of the target object, simulate based on the growth line of the target object, where the growth line represents the growth direction of the target object and is determined according to the slope and intercept of the target object.

[0097] The controller samples at a preset first angle based on the growth line of the target object to obtain a simulation result. Among them, the preset first angle is a preset angle range for the controller to sample based on the growth line, and the first preset angle can be set according to the actual usage scenario.

[0098] Optionally, sampling can be performed randomly at a preset first angle based on the target detection box and the growth line to obtain a simulation result. This is because the main trunk of the target object is within the target detection box. Therefore, sampling at a preset first angle based on the target detection box and the growth line can obtain a more accurate simulation result.

[0099] Optionally, in other embodiments, the first preset angle can be set according to the type of the target object, because the growth angles of different types of target objects are different.

[0100] Optionally, a regression model (OLS, Lasso, or Ransac regressor) can be used on the depth image of the target object to estimate the slope and intercept of the main trunk of the target object. By calculating the slope and intercept of the main trunk, the direction and position of the main trunk corresponding to the target can be determined, and then the growth line of the target object can be determined, so that the growth of the simulation result can be better simulated.

[0101] Figure 6 Schematic diagram of the slope and intercept of the target object in one embodiment, as Figure 6 shown, the straight line on the main trunk in the figure represents the growth line of the target object, and the growth direction of the target object can be obtained through the growth line.

[0102] After obtaining the depth image of the target object, starting from the key point in the depth image, a preset number of line segments are sampled at a preset second angle, where the preset second angle and the preset number are the preset angle range and number range for the controller to sample based on the key point, and the second preset angle and the preset number can be set according to the actual usage scenario.

[0103] Optionally, in other embodiments, the preset second angle and the preset number can be set according to the type of the target object, because the growth schedules and the thicknesses of the branches of different types of target objects are different.

[0104] Optionally, the line segments can also be sampled after setting a preset length according to the type of the target object, because the growth lengths of the branches of different target objects are different, so setting a preset length for the line segments can better simulate the simulation results.

[0105] Optionally, 10,000 (preset number) line segments of a specified length (set according to different plants) are sampled. The starting points of the line segments are along the key points, and the angles of the line segments are randomly sampled between the minimum and maximum angles in the horizontal direction (set according to the usual angle range of the stem of the plant to be recognized), so as to simulate the growth direction of the branches. Specifically, as shown in Figure 7 shown, Figure 7 is a schematic diagram of the simulation results in an embodiment. The shadows on both sides of the main trunk in the figure are the simulation results.

[0106] It should be noted that any one of the methods can be selected to simulate the branches of the target object, or two methods can be selected simultaneously to simulate the branches of the target object.

[0107] In the above embodiments, by simulating the growth direction of the target object in any of the above ways, the simulation results can be accurately obtained.

[0108] In one embodiment, the above simulation results are verified to obtain valid simulation results, including at least one of the following: calculating the depth missing rate of each simulation result; removing the simulation results with the depth missing rate exceeding the preset depth threshold to obtain valid simulation results; or projecting the depth image of the target object to obtain the depth image of the projected target object; in the depth image of the projected target object, calculating the inefficiency of each simulation result; removing the simulation results with the inefficiency exceeding the preset effective threshold to obtain valid simulation results.

[0109] Among them, an effective simulation result refers to a simulation result in which the line segments have corresponding depth values and the continuity degree of the line segments in the simulation result reaches a preset measurement standard (such as a preset first threshold and a preset second threshold), and the preset measurement standard is set according to the actual usage scenario. Among them, the depth missing rate is one of the criteria for measuring whether the simulation result is effective, and is calculated based on the missing proportion of the depth values of the line segments that make up the simulation result; the inefficiency is also one of the criteria for measuring whether the simulation result is effective, and is calculated based on the invalid proportion of the line segments that make up the simulation result.

[0110] Specifically, after obtaining the simulation result, the simulation result will be verified to obtain an effective simulation result.

[0111] Optionally, the controller checks whether there are valid depth values for the line segments that make up each simulation result in the depth map. When it is detected that there are missing depth values for the line segments that make up the simulation result in the depth map, the depth missing rate of the simulation result is calculated, that is, the depth missing rate of the line segments that make up the simulation result is statistically calculated. For example, the number of line segments with missing depth values in the current simulation result can be divided by the number of all line segments in the current simulation result to obtain the depth missing rate of the current simulation result. Then, the simulation results with a depth missing rate exceeding the preset first threshold are filtered out to obtain effective simulation results. Specifically combined with Figure 8 as shown Figure 8 is a schematic diagram of the simulation result after filtering out the missing depth values in an embodiment.

[0112] In other embodiments, the depth missing rate of the simulation result can be obtained by other calculation methods, and no specific limitation is made in this embodiment.

[0113] Optionally, project the depth map of the target object to obtain the projected depth image. For example, the depth image of the target object can be projected onto the x-z plane to obtain the depth image on the x-z plane, that is, the projected depth image. Then, screen the simulation results in the projected depth image. Specifically, sample the line segments on the plane where each simulation result is located after projection, and calculate the inefficiency of each simulation result. The controller checks whether the depth values of the sampled line segments are continuous, and calculates the proportion of the line segments with discontinuous depth values in all the line segments in the simulation result as the inefficiency of the simulation result, and removes the simulation results with an inefficiency exceeding the second preset threshold.

[0114] Exemplarily, under normal circumstances, the depth values of the sampled line segments should be continuous. For example, 10, 12, 14. If the sampled line segments are discontinuous, the inefficiency in the simulation result is calculated, and the calculation method of the inefficiency can be the same as that of the depth missing rate. When the inefficiency exceeds the preset second threshold, the corresponding simulation result is removed.

[0115] Optionally, specifically in combination with Figure 9 as shown Figure 9 is a schematic diagram of depth image projection in an embodiment. Since the depth map contains information about the three-dimensional structure, in this embodiment, the depth map is projected onto the x-z plane to filter out small leaf branches that are blocked in the front. For each remaining line (the simulation result after deleting the simulation results with a depth value missing rate reaching the threshold), sample internally along this line (in the x-z plane) at fixed intervals, and check whether the interval points have valid points in the projected depth image (i.e., whether the depth values are continuous), and delete the line segments with a percentage of invalid points greater than the preset percentage.

[0116] It should be noted that the controller can select any of the above methods to verify the simulation results, and can also use the above two methods simultaneously to verify the simulation results, that is, use multi-threading to verify the simulation results simultaneously, or can combine the two methods in any order. The specific combination order is not specifically limited in this embodiment.

[0117] In the above embodiment, through the above two verification methods, the validity of the simulation results can be verified to obtain valid simulation results.

[0118] In an embodiment, after obtaining the simulation results of the growth directions of the above simulation key points for the target object, it further includes: calculating the simulation results to obtain the maturity status of each simulation result.

[0119] Among them, the maturity status refers to the state of whether the simulated branch is mature or immature. The maturity status of the simulation results can be calculated according to the preset maturity range.

[0120] In an embodiment, after obtaining the valid simulation results, it further includes: comparing each valid simulation result to obtain the maturity status of each valid simulation result.

[0121] Specifically, after obtaining the valid simulation results, compare each simulation result to obtain the maturity status of each valid simulation result, where the maturity status is used to characterize whether the simulation result is mature. Specifically, the valid angle ranges of the valid simulation results can be compared. Exemplarily, if the valid line segment detected along a key point at the bottom of a stem is within 10 - 15°, and the valid line segment range detected at another key point is within 10 - 20°, and the maturity range in this embodiment is 16° - 20°, then the simulation result corresponding to the second key point is considered to be in a mature state.

[0122] Optionally, the projection depth map in the x-z plane can be continuously used. For each simulation result, in this embodiment, it is ensured that each line is within the preset mature angle range or the effective length range of the preset depth to distinguish the maturity of the branches, ensuring that small branches can continue to grow while large branches that have matured are detected.

[0123] Optionally, the controller can preset a mature range. Exemplarily, if the effective line segment in the effective simulation result is within 10 - 15°, and the preset mature range is set to be above 12°, then it can be determined that the simulation result is mature. Among them, the preset mature range can be set according to the actual application scenario. In this embodiment, the mature range is only for exemplary illustration and is not the real mature range of the target object.

[0124] In the above embodiment, by comparing each simulation result, the mature state of each simulation result is obtained to ensure that small branches can continue to grow while large branches that have matured are detected.

[0125] In one embodiment, obtaining the pose of the simulation result of the target object according to the valid simulation result above includes: screening out the target simulation result from the valid simulation results; calculating the pose according to the target simulation result.

[0126] The controller can screen out the target simulation result from the simulation results of the target object. The target simulation result refers to the branch that is selected from the simulation results for subsequent picking by the terminal, and then calculates the pose according to the target simulation result. Optionally, the pose can be calculated by 6dpose (a pose calculation method).

[0127] In one embodiment, screening out the target simulation result from the valid simulation results above includes: respectively calculating the angles between the simulation result and the camera plane, and between the simulation result and the ground; screening according to the angle between the simulation result and the camera plane and the angle between the simulation result and the ground to obtain the target simulation result.

[0128] The controller can respectively calculate the angles between the simulation result and the camera plane, and between the simulation result and the ground plane. The camera plane refers to the plane where the camera mirror is located, and then screens according to the angle between the simulation result and the camera plane and the angle between the simulation result and the ground to obtain the target simulation result.

[0129] Optionally, the controller can first screen out candidate simulation results from the simulation results. The candidate simulation results refer to the simulation results whose angles with the camera plane are within the first preset angle range and whose angles with the horizontal ground are within the second preset angle range. These simulation results are convenient for the harvesting robot to pick. Among them, the first preset angle range and the second preset angle range are the measurement criteria for judging candidate simulation results, and are actually set according to the actual usage scenario.

[0130] Optionally, the candidate simulation results can be calculated by separately calculating the angles between the simulation results and the camera plane and between the simulation results and the ground, and then filtering out the simulation results where the angle between the simulation result and the camera plane is not within the first preset angle range and the angle between the simulation result and the ground is not within the second preset angle range. This can ensure that the angles between the candidate simulation results and the camera plane are within the first preset angle range and the angles between the candidate simulation results and the horizontal ground plane are within the second preset angle range, which is convenient for the operation of the robot and the credibility of these simulation results is relatively high.

[0131] Optionally, the target simulation results are selected from the candidate simulation results, where the target simulation results can be the simulation results closest to the harvesting robot or the candidate simulation results with the largest angles with the camera plane and the horizontal ground.

[0132] In one embodiment, the unit vector from the branch point to the vertex of the simulation result is calculated. For each 3D line, the angle theta with the camera plane and the angle phi with the horizontal plane are calculated. Any simulation results not within the first preset angle range and the second preset angle range are filtered out. This method will ensure that the branches we are going to prune are within the first preset angle range with the camera plane and within the second preset angle range with the horizontal ground plane, which is convenient for the operation of the robot.

[0133] In the above embodiment, by separately calculating the angles between the simulation results and the camera plane and between the simulation results and the ground, and screening out the target simulation results based on the angles between the simulation results and the camera plane and between the simulation results and the ground, the target simulation results have angles with the camera and the ground, which can facilitate the operation of the harvesting robot and have a high recognition credibility.

[0134] In one embodiment, the process of plant simulated branch recognition is as follows:

[0135] 1. Obtain the RGBD image of the plant as Figure 3 shown.

[0136] 2. Input the RGBD image of the plant into the AI general detection model. The AI general detection model obtains the target detection box where the main trunk of the plant is located, thereby obtaining the main trunk information corresponding to the plant. Specifically, as shown in combination with Figure 4 shown, Figure 4 the black box in is the target detection box, and the information in the target detection box is the main trunk information of the plant.

[0137] 3. Take the main trunk information of the plant as the input and input it into the neural network for key point detection. Through the neural network for key point detection, the intersection points of the branches and the main trunk in the above plant are obtained. Hereinafter, this intersection point is called the key point, as shown in Figure 3The small hollow black circles in

[0138] 4. After obtaining at least one key point of the above plant, select the bottommost key point, and obtain the depth information corresponding to the bottommost key point from the depth image corresponding to the image to be processed, so as to obtain the depth of the main trunk of the above plant; then filter out the noise according to the set depth threshold to obtain the depth map of the complete plant, which can be specifically combined with Figure 5 as shown, where the set depth threshold is set according to the current plant.

[0139] 5. Use a regression model on the depth map of the above plant to estimate the slope and intercept of the main trunk of the above plant, and determine the growth line of the plant according to the slope and intercept, combined with Figure 6 as shown. Figure 6 The lines on the main trunk of the above plant in are the growth lines of the plant, and the growth lines represent the growth direction of the plant.

[0140] 6. Simulate the growth direction of the branches of the above plant to obtain simulated branches. The simulation process can include the following two methods: 1) Based on the main trunk of the above plant that has been simulated, sample 10,000 line segments of a specified length. Among them, the starting point of the line segment is along the included key point, and the angle of the line segment is randomly sampled between the minimum and maximum angles in the horizontal direction to obtain simulated branches, combined with Figure 7 as shown. Figure 7 The shaded parts on both sides of the main trunk in are the simulated branches obtained by simulating with the key point as the starting point; 2) Randomly sample based on the target box and the growth line, and the sampling angle is approximately in the range of -30 degrees to 60 degrees to obtain simulated branches.

[0141] 7. Verify the simulated branches to obtain effective branches. The verification process also includes two methods: 1) For each line segment that makes up each simulated branch, check the valid depth values of the line segments that make up each simulated branch on the depth map. Once the depth values of the line segments that make up each simulated branch are missing on the depth map, calculate the depth missing rate of the simulated branch, and remove the simulated branches with a depth missing rate exceeding the preset first threshold to obtain effective branches, which can be specifically combined with Figure 8 as shown. Figure 8 The branches in the shaded part in are the effective simulated branches after verification; 2) Since the depth map contains information about the three-dimensional structure, project the depth map of the plant onto the x-z plane to filter out the small leaf branches that are blocked in front. Sample the line segments on the plane where each simulated branch is located after projection, and check whether the interval points have valid points in the projected depth mask, and calculate the inefficiency. When the inefficiency of a certain simulated branch exceeds the preset second threshold, delete the simulated branch (this method will delete the lines with discontinuous depth in the x-z plane), which can be specifically combined with Figure 9 as shown.

[0142] 8. Calculate the maturity state of the effective simulated branches, and this step will continuously use the projected depth map in the x-z plane: for each simulated branch, ensure that each line is within a certain angular range and an effective length range in terms of depth, which is used to distinguish the maturity of the branches, ensuring that small branches can continue to grow while the already mature large branches are detected. For example, if the effective line detected along a key point at the bottom of a stem is within 10 - 15°, and the effective line range detected at another key point is within 10 - 20°, then the simulated branch corresponding to the second key point is considered thicker and more mature.

[0143] 9. Screen the target simulated branches. Since the screening was previously done on a 2D plane, in the remaining simulated branches, use the depth information to extract the 3D positions of the starting and ending points of the lines. Calculate the unit vector from the branch point to the vertex, that is, the unit vector from the key point to the end of the branch. For each 3D line, calculate the angle theta with the camera plane and the angle phi with the horizontal plane. Filter out the lines that are not within the first preset angle range and the second preset angle range. This method will ensure that the target simulated branches have an angle with the camera plane and the angle is within the first preset angle range, and the target simulated branches have an angle with the horizontal ground plane and the angle is within the second preset angle range, which is convenient for the operation of the robot. In addition, since the target simulated branches have an angle with the camera plane and the horizontal ground, the credibility of the target simulated branches is higher. It should be noted that this step is an optional step.

[0144] 10. Among the effective simulated branches, screen out the target simulated branches. For example, select the simulated branch closest to the robot, and then calculate the pose of the target simulated branch.

[0145] In the above embodiments, by integrating the AI detection model and the visual solution of the simulation results, the specific position of each branch can be accurately located, which is convenient for subsequent picking.

[0146] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0147] Based on the same inventive concept, an embodiment of the present application further provides a branch recognition device for implementing the above-mentioned branch recognition method. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the branch recognition device provided below can refer to the limitations on the branch recognition method in the above text, and will not be repeated here.

[0148] In one embodiment, as Figure 10 shown, a branch recognition device is provided, including: an acquisition module 100, a key point acquisition module 200, a simulation module 300, and a calculation module 400, where:

[0149] The acquisition module 100 is configured to acquire an image to be processed.

[0150] The key point acquisition module 200 obtains the key points of the target object according to the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object.

[0151] The simulation module 300 is configured to simulate the growth direction of the branches of the target object based on the key points to obtain a simulation result, and verify the simulation result to obtain a valid simulation result.

[0152] The calculation module 400 is configured to obtain the pose of the simulation result of the target object according to the valid simulation result.

[0153] In one embodiment, the above key point acquisition module 200 includes:

[0154] The detection sub-module is configured to detect the image to be processed to obtain the main trunk information of the target object.

[0155] The recognition sub-module is configured to recognize the main trunk information of the target object to obtain the key points.

[0156] In one embodiment, the above simulation module 300 includes:

[0157] The depth image acquisition sub-module obtains the depth image of the target object based on the key points.

[0158] The simulation result sub-module is configured to simulate the growth direction of the branches of the target object in the depth image of the target object to obtain a simulation result.

[0159] The verification sub-module is configured to verify the simulation result to obtain a valid simulation result.

[0160] In one embodiment, the above depth image acquisition sub-module includes:

[0161] A depth information acquisition unit, configured to acquire a depth image corresponding to the image to be processed, and obtain the depth information of key points based on the depth image corresponding to the image to be processed.

[0162] A denoising unit, configured to filter out the noise in the depth image corresponding to the image to be processed based on the depth information of the key points and according to a set depth threshold, so as to obtain the depth image of the target object.

[0163] In one embodiment, the above-mentioned simulation result sub-module further includes:

[0164] A first sampling unit, configured to sample in the depth image of the target object at a preset first angle based on the growth line of the target object, so as to obtain a simulation result; the growth line is determined according to the slope and intercept corresponding to the target object.

[0165] A second sampling unit, configured to sample in the depth image of the target object starting from the key point, with a preset number of line segments and a preset second angle, so as to obtain a simulation result.

[0166] In one embodiment, the above-mentioned verification sub-module further includes:

[0167] A depth missing rate calculation unit, configured to calculate the depth missing rate of each simulation result.

[0168] A first filtering unit, configured to remove the simulation results with a depth missing rate exceeding a preset first threshold, so as to obtain valid simulation results.

[0169] A projection unit, configured to project the depth image of the target object to obtain the projected depth image of the target object.

[0170] An inefficiency calculation unit, configured to calculate the inefficiency of each simulation result in the projected depth image of the target object.

[0171] A second filtering unit, configured to remove the simulation results with an inefficiency exceeding a preset second threshold, so as to obtain valid simulation results.

[0172] In one embodiment, the above-mentioned device further includes:

[0173] A maturity state acquisition module, configured to compare each valid simulation result to obtain the maturity state of each valid simulation result.

[0174] In one embodiment, the above-mentioned calculation module 400 includes:

[0175] A target simulation result screening sub-module, configured to screen out the target simulation result from the valid simulation results.

[0176] A pose calculation sub-module, configured to calculate the pose according to the target simulation result.

[0177] In one embodiment, the above-mentioned target simulation result screening sub-module includes:

[0178] An angle calculation unit for calculating the angles between the simulation result and the camera plane, and between the simulation result and the ground, respectively.

[0179] A target simulation result selection unit for screening according to the angles between the simulation result and the camera plane, and between the simulation result and the ground to obtain the target simulation result.

[0180] Each module in the above-mentioned branch recognition device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0181] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 11 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the image data to be processed. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a branch recognition method.

[0182] Those skilled in the art can understand that Figure 11 the structure shown in

[0183] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0184] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements the steps of the method in any one of the above embodiments.

[0185] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps of the method in any one of the above embodiments.

[0186] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, a database, or other media used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAMs), magnetoresistive random access memories (MRAMs), ferroelectric random access memories (FRAMs), phase change memories (PCMs), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0187] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0188] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A branch recognition method, characterized in that, The method includes: Obtain an image to be processed; Obtain key points of a target object according to the image to be processed, where the target object is located in the image to be processed, and the key points represent the intersection points of the branches and the main trunk of the target object; Based on the key points, simulate the growth direction of the branches of the target object to obtain a simulation result, and verify the simulation result to obtain a valid simulation result; According to the valid simulation result, obtain the pose of the simulation result of the target object; The step of based on the key points, simulating the growth direction of the branches of the target object to obtain a simulation result, and verifying the simulation result to obtain the valid simulation result includes: Based on the key points, obtain the depth image of the target object; In the depth image of the target object, simulate the growth direction of the branches of the target object to obtain a simulation result; Verify the simulation result to obtain the valid simulation result; The step of in the depth image of the target object, simulating the growth direction of the branches of the target object to obtain a simulation result includes at least one of the following: In the depth image of the target object, sample based on the growth line of the target object at a preset first angle to obtain the simulation result; the growth line is determined according to the slope and intercept corresponding to the target object; or In the depth image of the target object, starting from the key points, sample with a preset number of line segments and a preset second angle to obtain the simulation result.

2. The method according to claim 1, characterized in that, The step of obtaining key points of a target object according to the image to be processed includes: Detect the image to be processed to obtain the main trunk information of the target object; Identify the main trunk information of the target object to obtain the key points.

3. The method according to claim 1, characterized in that, The step of based on the key points, obtaining the depth image of the target object includes: Obtain the depth image corresponding to the image to be processed, and obtain the depth information of the key points according to the depth image corresponding to the image to be processed; Based on the depth information of the key points, and according to a set depth threshold, filter out the noise in the depth image corresponding to the image to be processed to obtain the depth image of the target object.

4. The method according to claim 1, wherein The step of verifying the simulation result to obtain the valid simulation result includes at least one of the following: Calculate the depth missing rate of each simulation result; Remove the simulation results with the depth missing rate exceeding a preset first threshold to obtain the valid simulation results; Or Project the depth image of the target object to obtain the projected depth image of the target object; In the projected depth image of the target object, calculate the inefficiency of each simulation result; Remove the simulation results with the inefficiency exceeding a preset second threshold to obtain the valid simulation results.

5. The method according to claim 1, wherein After obtaining the valid simulation results, it further includes: Compare each valid simulation result to obtain the maturity state of each valid simulation result.

6. The method according to claim 1, characterized in that, The step of according to the valid simulation result, obtaining the pose of the simulation result of the target object includes: Select a target simulation result from the valid simulation results; Calculate the pose according to the target simulation result.

7. The method according to claim 6, wherein Filter out the target simulation result from the valid simulation results, including: Calculate the angles between the simulation result and the camera plane, and between the simulation result and the ground respectively; Filter according to the angle between the simulation result and the camera plane and the angle between the simulation result and the ground to obtain the target simulation result.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images