Plant height determination method, device, electronic equipment and medium

Abstract

The present application relates to image analysis technology, and the present application provides a method, device, electronic device and medium for determining plant height. The method obtains a color image and a depth image of a plant to be detected, maps the color image and the depth image to obtain a target image, detects the color image using a pre-trained MobileNet-SSD network, obtains a detection frame with the plant to be detected, extracts the target contour of the plant to be detected from the detection frame, determines the depth values ​​of all pixels in the target contour according to the target image, performs denoising on all the depth values ​​to obtain a target depth value, and determines the height of the plant to be detected according to the target depth value. The present application can improve the accuracy of determining the height of the plant to be detected.

Description

Technical Field

[0001] The present application relates to the field of image analysis technology, and in particular to a method, device, electronic equipment and medium for determining plant height. Background Art

[0002] At present, analyzing the daily growth of plants is helpful to determine the best way to plant them, so as to improve the yield and quality of plants, thereby reducing the cost of planting and bringing good news to growers. The existing method is to detect the bounding box with the plant to be detected through a deep learning algorithm, and then determine the height of the plant to be detected based on the depth value in the bounding box. However, since the bounding box is usually a quadrilateral, and the plant to be detected is not a quadrilateral, some irrelevant information will appear in the detected bounding box, such as leaves, weeds, etc., resulting in inaccurate calculation of the height of the plant to be detected. Summary of the invention

[0003] In view of the above, it is necessary to provide a method, device, electronic device and medium for determining plant height, which can improve the accuracy of plant height.

[0004] A method for determining plant height, the method comprising:

[0005] Acquire a color image and a depth image of the plant to be detected;

[0006] Mapping the color image and the depth image to obtain a target image;

[0007] Using a pre-trained mobilenet-ssd network to detect the color image, and obtain a detection frame with the plant to be detected;

[0008] Extracting the target outline of the plant to be detected from the detection frame;

[0009] Determine the depth values ​​of all pixels in the target contour according to the target image;

[0010] De-noising is performed on all the depth values ​​to obtain a target depth value, and the height of the plant to be detected is determined according to the target depth value.

[0011] According to an optional embodiment of the present application, the acquisition of the color image and the depth image of the plant to be detected includes a combination of one or more of the following methods:

[0012] Using a first lens of a camera device to photograph the plant to be detected to obtain the color image, and using a second lens of the camera device to photograph the plant to be detected to obtain the depth image; and / or

[0013] The label of the plant to be detected is determined, and an image with the label is obtained from a first configuration library as the color image, and an image with the label is obtained from a second configuration library as the depth image.

[0014] According to an optional embodiment of the present application, the denoising process is performed on all the depth values ​​to obtain a target depth value, and the height of the plant to be detected is determined according to the target depth value, including:

[0015] Acquire a depth value equal to a preset value from all the depth values, and determine the acquired depth value as a zero value;

[0016] Deleting the zero value from all the depth values ​​and determining the remaining depth value as the target depth value;

[0017] determining the number of target depth values;

[0018] Calculating the sum of the target depth values;

[0019] Dividing the sum by the number to obtain the distance between the plant to be detected and the camera device;

[0020] Determining the camera height of the camera device;

[0021] The camera height is subtracted from the distance to obtain the height of the plant to be detected.

[0022] According to an optional embodiment of the present application, mapping the color image and the depth image to obtain a target image includes:

[0023] Obtaining all depth pixels on the depth image;

[0024] Mapping all the depth pixels into a preset depth coordinate system to obtain depth coordinates of all the depth pixels;

[0025] Determine the world coordinates of all the depth pixels according to all the depth coordinates and a preset world coordinate system;

[0026] Determine the positions of all the depth pixels on the color image according to all the world coordinates, and determine the color pixels at the positions on the color image;

[0027] Each depth pixel is fused with each color pixel to obtain the target image.

[0028] According to an optional embodiment of the present application, the method of detecting the color image using a pre-trained mobilenet-ssd network to obtain a detection frame with the plant to be detected includes:

[0029] Obtain the depth convolution kernel and the point convolution kernel in the mobilenet-ssd network;

[0030] Extracting features of the color image using the deep convolution kernel to obtain a first feature map;

[0031] Processing the first feature map using the point convolution kernel to obtain a second feature map;

[0032] The detection frame is determined according to the second feature map.

[0033] According to an optional embodiment of the present application, extracting the target contour of the plant to be detected from the detection frame includes:

[0034] Deleting the background image in the detection frame to obtain a grayscale image;

[0035] The outline of the plant to be detected is detected on the grayscale image to obtain the target outline.

[0036] According to an optional embodiment of the present application, determining the depth values ​​of all pixels in the target contour according to the target image includes:

[0037] Determine the target position of each pixel on the target image;

[0038] A depth value at the target position is obtained from the target image as the depth value of each pixel.

[0039] A plant height determination device, the plant height determination device comprising:

[0040] An acquisition unit, used for acquiring a color image and a depth image of the plant to be detected;

[0041] A mapping unit, used for mapping the color image and the depth image to obtain a target image;

[0042] A detection unit, used to detect the color image using a pre-trained mobilenet-ssd network to obtain a detection frame with the plant to be detected;

[0043] An extraction unit, used for extracting the target outline of the plant to be detected from the detection frame;

[0044] A determination unit, configured to determine depth values ​​of all pixels in the target contour according to the target image;

[0045] The determination unit is further used to perform denoising on all the depth values ​​to obtain a target depth value, and determine the height of the plant to be detected according to the target depth value.

[0046] According to an optional embodiment of the present application, the determining unit performs denoising on all the depth values ​​to obtain a target depth value, and determines the height of the plant to be detected according to the target depth value, including:

[0047] Acquire a depth value equal to a preset value from all the depth values, and determine the acquired depth value as a zero value;

[0048] Deleting the zero value from all the depth values ​​and determining the remaining depth value as the target depth value;

[0049] determining the number of target depth values;

[0050] Calculating the sum of the target depth values;

[0051] Dividing the sum of the target depth values ​​by the number to obtain the distance between the plant to be detected and the camera device;

[0052] Determining the camera height of the camera device;

[0053] The camera height is subtracted from the distance to obtain the height of the plant to be detected.

[0054] According to an optional embodiment of the present application, the acquisition unit is specifically used for:

[0055] Using a first lens of a camera device to photograph the plant to be detected to obtain the color image, and using a second lens of the camera device to photograph the plant to be detected to obtain the depth image; and / or

[0056] The label of the plant to be detected is determined, and an image with the label is obtained from a first configuration library as the color image, and an image with the label is obtained from a second configuration library as the depth image.

[0057] According to an optional embodiment of the present application, the mapping unit is specifically used for:

[0058] Obtaining all depth pixels on the depth image;

[0059] Mapping all the depth pixels into a preset depth coordinate system to obtain depth coordinates of all the depth pixels;

[0060] Determine the world coordinates of all the depth pixels according to all the depth coordinates and a preset world coordinate system;

[0061] Determine the positions of all the depth pixels on the color image according to all the world coordinates, and determine the color pixels at the positions on the color image;

[0062] Each depth pixel is fused with each color pixel to obtain the target image.

[0063] According to an optional embodiment of the present application, the detection unit is specifically used for:

[0064] Obtain the depth convolution kernel and the point convolution kernel in the mobilenet-ssd network;

[0065] Extracting features of the color image using the deep convolution kernel to obtain a first feature map;

[0066] Processing the first feature map using the point convolution kernel to obtain a second feature map;

[0067] The detection frame is determined according to the second feature map.

[0068] According to an optional embodiment of the present application, the extraction unit is specifically used for:

[0069] Deleting the background image in the detection frame to obtain a grayscale image;

[0070] The outline of the plant to be detected is detected on the grayscale image to obtain the target outline.

[0071] According to an optional embodiment of the present application, the determining unit determines the depth values ​​of all pixels in the target contour according to the target image, including:

[0072] Determine the target position of each pixel on the target image;

[0073] A depth value at the target position is obtained from the target image as the depth value of each pixel.

[0074] An electronic device, comprising:

[0075] a memory storing at least one instruction; and

[0076] A processor executes the instructions stored in the memory to implement the plant height determination method.

[0077] A computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the plant height determination method.

[0078] It can be seen from the above technical scheme that the present application can obtain a target image containing a depth value by mapping the color image and the depth image, and then the detection frame can be quickly detected through a pre-trained Mobilenet-SSD network to improve the detection efficiency, and the target contour of the plant to be detected is extracted from the detection frame, and the target contour excluding irrelevant information can be extracted. By determining the depth values ​​of all pixels in the target contour according to the target image, the depth values ​​of all pixels can be accurately and quickly determined, and then by denoising all the depth values ​​to obtain the target depth value, it can be ensured again that there is no depth value containing irrelevant information in the target depth value, and the height of the plant to be detected can be accurately determined according to the target depth value. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is an application environment diagram of a preferred embodiment of the plant height determination method of the present application.

[0080] Figure 2 It is a flow chart of a preferred embodiment of the method for determining plant height of the present application.

[0081] Figure 3 It is a functional module diagram of a preferred embodiment of the plant height determination device of the present application.

[0082] Figure 4 It is a schematic diagram of the structure of an electronic device of a preferred embodiment of the plant height determination method of the present application. DETAILED DESCRIPTION

[0083] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0084] like Figure 1 , which is an application environment diagram of a preferred embodiment of the plant height determination method of the present application. The camera device 2 communicates with the electronic device 1, and the camera device 2 includes a first lens 20 and a second lens 21. The first lens 20 can capture color images, and the second lens 21 can capture depth images.

[0085] like Figure 2 FIG. 1 is a flow chart of a preferred embodiment of the plant height determination method of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0086] The plant height determination method is applied to one or more electronic devices 1, and the electronic device 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital signal processors (Digital Signal Processor, DSP), embedded devices, etc.

[0087] The electronic device 1 can be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0088] The electronic device 1 may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0089] The network where the electronic device 1 is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0090] S10, obtaining a color image and a depth image of the plant to be detected.

[0091] In at least one embodiment of the present application, the color image refers to an RGB three-channel color image, and the depth image refers to an image that uses the distance from the image collector to each point in the scene as a pixel value.

[0092] In at least one embodiment of the present application, the color image and the depth image may be acquired from a camera device or from a configuration library.

[0093] Furthermore, both the color image and the depth image contain the plant to be detected.

[0094] In at least one embodiment of the present application, the electronic device acquires the color image and the depth image of the plant to be detected by one or more of the following methods:

[0095] (1) The electronic device uses the first lens of the camera device to photograph the plant to be detected to obtain the color image, and uses the second lens of the camera device to photograph the plant to be detected to obtain the depth image.

[0096] The imaging device includes dual lenses, namely the first lens and the second lens. Furthermore, the imaging device may be a camera, and the imaging device may be installed just above the detected plant for convenient photographing.

[0097] Furthermore, the plant to be detected can be any plant whose daily growth needs to be analyzed, such as rose, sunflower, rice, etc.

[0098] Specifically, when it is detected that the plant to be detected is directly in front of the camera device, the electronic device turns on the camera device and uses the first lens to photograph the plant to be detected to obtain the color image; further, the electronic device uses the second lens to photograph the plant to be detected to obtain the depth image.

[0099] Through the above implementation, a color image and a depth image containing the plant to be detected can be quickly acquired.

[0100] (2) The electronic device determines the label of the plant to be detected. Further, the electronic device obtains an image with the label from a first configuration library as the color image. Further, the electronic device obtains an image with the label from a second configuration library as the depth image.

[0101] The first configuration library stores mapping relationships between a plurality of color images and the labels, and the second configuration library stores mapping relationships between a plurality of depth images and the labels.

[0102] Furthermore, the label may be the serial number of the plant to be detected, for example, the label may be 0001.

[0103] Through the mapping relationship between the label and the color image, the color image can be accurately acquired, and further through the mapping relationship between the label and the depth image, the depth image can be accurately acquired.

[0104] S11, mapping the color image and the depth image to obtain a target image.

[0105] In at least one embodiment of the present application, the target image refers to an image generated by fusing pixels of the color image and pixels of the depth image.

[0106] In at least one embodiment of the present application, the electronic device performs mapping processing on the color image and the depth image to obtain a target image, including:

[0107] The electronic device obtains all depth pixels on the depth image. Further, the electronic device maps all the depth pixels to a preset depth coordinate system to obtain depth coordinates of all the depth pixels. Further, the electronic device determines the world coordinates of all the depth pixels based on all the depth coordinates and a preset world coordinate system. Further, the electronic device determines the positions of all the depth pixels on the color image based on all the world coordinates, and determines the color pixels at the positions on the color image. Further, the electronic device fuses each depth pixel with each color pixel to obtain the target image.

[0108] Among them, the preset depth coordinate system and the preset world coordinate system can be obtained from an open source system, or can be arbitrarily set by the user according to the application scenario, and this application does not impose any restrictions on this.

[0109] Through the above implementation, a target image including a depth value can be generated, so as to subsequently determine the height of the plant to be detected.

[0110] S12, using a pre-trained mobilenet-ssd network to detect the color image and obtain a detection frame with the plant to be detected.

[0111] In at least one embodiment of the present application, the frame to be detected is obtained by performing feature extraction on the color image using a convolution kernel in the mobilenet-ssd network.

[0112] In at least one embodiment of the present application, before using a pre-trained mobilenet-ssd network to detect the color image and obtain a detection frame with the plant to be detected, the method further includes:

[0113] The electronic device determines the field to which the plant to be detected belongs. Further, the electronic device obtains multiple color training images on the field. Further, the electronic device fuses the multiple color training images obtained to obtain a data set. The electronic device divides the data set into a training set and a validation set. Further, the electronic device uses the color training images in the training set to train an open source convolutional network to obtain a learner. Further, the electronic device uses the color training images in the validation set to adjust the learner to obtain the MobileNet-SSD network.

[0114] In at least one embodiment of the present application, before dividing the data set into a training set and a validation set, the method further includes:

[0115] The electronic device calculates the number of color training images in the data set, and when the number is less than a preset number, the electronic device increases the number of color training images in the data set by using a data enhancement algorithm.

[0116] Through the above implementation, it is possible to avoid the poor generalization ability of the trained Mobilenet-SSD network due to insufficient number of color training images.

[0117] In at least one embodiment of the present application, the electronic device dividing the data set into a training set and a validation set includes:

[0118] The electronic device randomly divides the data set into at least one data packet according to a preset ratio, determines any one of the at least one data packet as the verification set, and determines the remaining data packets as the training set, and repeats the above steps until all the data packets are used as the verification set in turn.

[0119] Among them, the preset ratio can be customized and is not limited in this application.

[0120] By dividing the data set through the above implementation method, each color training image in the data set participates in training and verification, thereby improving the fitness of the training of the mobilenet-ssd network.

[0121] In at least one embodiment of the present application, the electronic device uses the color training images in the verification set to adjust the learner, and obtaining the mobilenet-ssd network includes:

[0122] The electronic device uses a hyperparameter grid search method to determine the optimal hyperparameter point from the verification set. Further, the electronic device adjusts the learner through the optimal hyperparameter point to obtain the mobilenet-ssd network.

[0123] Specifically, the electronic device splits the verification set according to a fixed step size to obtain a target subset, traverses the parameters of the two end points on the target subset, verifies the learner through the parameters of the two end points, obtains the learning rate of each parameter, determines the parameter with the best learning rate as the first hyperparameter point, and within the neighborhood of the first hyperparameter point, reduces the step size and continues to traverse until the step size is a preset step size, that is, the obtained hyperparameter point is the optimal hyperparameter point. Furthermore, the electronic device adjusts the learner according to the optimal hyperparameter point to obtain the mobilenet-ssd network.

[0124] Among them, this application does not limit the preset step size.

[0125] Through the above implementation, the mobilenet-ssd network can be made more suitable for the detection of color images in the field.

[0126] In at least one embodiment of the present application, the electronic device detects the color image using a pre-trained mobilenet-ssd network, and obtaining a detection frame with the plant to be detected includes:

[0127] The electronic device obtains the deep convolution kernel and the point convolution kernel in the mobilenet-ssd network. Further, the electronic device uses the deep convolution kernel to extract the features of the color image to obtain a first feature map. Further, the electronic device uses the point convolution kernel to process the first feature map to obtain the detection frame.

[0128] Among them, the depth convolution kernel can be a 16*16*128 matrix, and further, the point convolution kernel can be a 1*1*16 matrix.

[0129] The detection frame can be quickly detected through the pre-trained mobilenet-ssd network, thereby improving the detection efficiency.

[0130] S13, extracting the target outline of the plant to be detected from the detection frame.

[0131] In at least one embodiment of the present application, the target contour refers to a contour after irrelevant information is removed from the detection frame, and the shape of the target contour is determined according to the shape of the plant to be detected.

[0132] In at least one embodiment of the present application, the electronic device extracting the target outline of the plant to be detected from the detection frame includes:

[0133] The electronic device deletes the background image in the detection frame to obtain a grayscale image. Further, the electronic device detects the outline of the plant to be detected on the grayscale image to obtain the target outline.

[0134] By deleting the background image, the interference caused by the background image can be avoided, thereby improving the accuracy of extracting the target contour.

[0135] S14: Determine depth values ​​of all pixels in the target contour according to the target image.

[0136] In at least one embodiment of the present application, the depth value refers to the height of a feature point on the plant to be detected corresponding to a pixel from a camera device.

[0137] In at least one embodiment of the present application, the electronic device determining the depth values ​​of all pixels in the target contour according to the target image includes:

[0138] The electronic device determines a target position of each pixel point on the target image. Further, the electronic device obtains a depth value at the target position from the target image as a depth value of each pixel point.

[0139] By determining the depth values ​​of all the pixels in the target contour according to the target image, the depth values ​​of all the pixels can be determined accurately and quickly.

[0140] S15, performing denoising processing on all the depth values ​​to obtain a target depth value, and determining the height of the plant to be detected according to the target depth value.

[0141] In at least one embodiment of the present application, the electronic device performs denoising on all the depth values ​​to obtain a target depth value, and determines the height of the plant to be detected according to the target depth value, including:

[0142] The electronic device obtains a depth value equal to a preset value from all the depth values, and determines the obtained depth value as a zero value. Further, the electronic device deletes the zero value from all the depth values, and determines the remaining depth value as the target depth value. Further, the electronic device determines the number of the target depth values, calculates the sum of the target depth values, and divides the sum by the number to obtain the distance between the plant to be detected and the camera device. Further, the electronic device determines the camera height of the camera device, and the electronic device subtracts the camera height from the distance to obtain the height of the plant to be detected.

[0143] By performing denoising on all the depth values ​​to obtain a target depth value, it is possible to again ensure that the target depth value does not contain a depth value with irrelevant information, and the height of the plant to be detected can be accurately determined based on the target depth value.

[0144] In at least one embodiment of the present application, after determining the height of the plant to be detected according to the target depth value, the plant height determination method further includes:

[0145] When the height is less than a preset height, the electronic device generates an alarm message according to the height. Further, the electronic device encrypts the alarm message using a symmetric encryption algorithm to obtain a ciphertext. Further, the electronic device determines an alarm level of the ciphertext according to the plant to be detected. The electronic device determines an alarm mode according to the alarm level. Further, the electronic device sends the ciphertext in the alarm mode.

[0146] Among them, the preset height can be set according to the expected growth rate of the plant to be detected, and the present application does not limit the value of the preset height.

[0147] Furthermore, the alarm levels include: level one, level two, etc.

[0148] Furthermore, the alarm method includes: an alarm sound from a speaker, an email, a telephone call, etc.

[0149] Through the above implementation, an alarm message can be issued when the height is less than the preset height. In addition, by encrypting the alarm message, it is possible to prevent the alarm message from being tampered with, thereby improving the security of the alarm message. At the same time, the alarm mode is determined according to the alarm level, and the alarm message can be sent in an appropriate alarm mode, making the sending of the alarm message more humane.

[0150] It can be seen from the above technical scheme that the present application can obtain a target image containing a depth value by mapping the color image and the depth image, and then the detection frame can be quickly detected through a pre-trained Mobilenet-SSD network to improve the detection efficiency, and the target contour of the plant to be detected is extracted from the detection frame, and the target contour excluding irrelevant information can be extracted. By determining the depth values ​​of all pixels in the target contour according to the target image, the depth values ​​of all pixels can be accurately and quickly determined, and then by denoising all the depth values ​​to obtain the target depth value, it can be ensured again that there is no depth value containing irrelevant information in the target depth value, and the height of the plant to be detected can be accurately determined according to the target depth value.

[0151] like Figure 3As shown, it is a functional module diagram of a preferred embodiment of the plant height determination device of the present application. The plant height determination device 11 includes an acquisition unit 110, a mapping unit 111, a detection unit 112, an extraction unit 113, a determination unit 114, a fusion unit 115, a division unit 116, a training unit 117, an adjustment unit 118, a calculation unit 119, an enhancement unit 120, a generation unit 121, an encryption unit 122 and a sending unit 123. The module / unit referred to in the present application refers to a series of computer program segments that can be executed by the processor 13 and can complete fixed functions, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0152] The acquisition unit 110 acquires a color image and a depth image of the plant to be detected.

[0153] In at least one embodiment of the present application, the color image refers to an RGB three-channel color image, and the depth image refers to an image that uses the distance from the image collector to each point in the scene as a pixel value.

[0154] In at least one embodiment of the present application, the color image and the depth image may be acquired from a camera device or from a configuration library.

[0155] Furthermore, both the color image and the depth image contain the plant to be detected.

[0156] In at least one embodiment of the present application, the acquisition unit 110 acquires the color image and the depth image of the plant to be detected by one or more of the following methods:

[0157] (1) The acquisition unit 110 uses the first lens of the camera device to photograph the plant to be detected to obtain the color image, and uses the second lens of the camera device to photograph the plant to be detected to obtain the depth image.

[0158] The imaging device includes dual lenses, namely the first lens and the second lens. Furthermore, the imaging device may be a camera, and the imaging device may be installed just above the detected plant for convenient photographing.

[0159] Furthermore, the plant to be detected can be any plant whose daily growth needs to be analyzed, such as rose, sunflower, rice, etc.

[0160] Specifically, when it is detected that the plant to be detected is directly in front of the camera device, the acquisition unit 110 turns on the camera device and uses the first lens to photograph the plant to be detected to obtain the color image; further, the electronic device uses the second lens to photograph the plant to be detected to obtain the depth image.

[0161] Through the above implementation, a color image and a depth image containing the plant to be detected can be quickly acquired.

[0162] (2) The acquisition unit 110 determines the label of the plant to be detected. Further, the acquisition unit 110 acquires an image with the label from a first configuration library as the color image. Further, the acquisition unit 110 acquires an image with the label from a second configuration library as the depth image.

[0163] The first configuration library stores mapping relationships between a plurality of color images and the labels, and the second configuration library stores mapping relationships between a plurality of depth images and the labels.

[0164] Furthermore, the label may be the serial number of the plant to be detected, for example, the label may be 0001.

[0165] Through the mapping relationship between the label and the color image, the color image can be accurately acquired, and further through the mapping relationship between the label and the depth image, the depth image can be accurately acquired.

[0166] The mapping unit 111 performs mapping processing on the color image and the depth image to obtain a target image.

[0167] In at least one embodiment of the present application, the target image refers to an image generated by fusing pixels of the color image and pixels of the depth image.

[0168] In at least one embodiment of the present application, the mapping unit 111 performs mapping processing on the color image and the depth image to obtain a target image including:

[0169] The mapping unit 111 obtains all depth pixels on the depth image. Further, the mapping unit 111 maps all the depth pixels to a preset depth coordinate system to obtain depth coordinates of all the depth pixels. Further, the mapping unit 111 determines the world coordinates of all the depth pixels based on all the depth coordinates and a preset world coordinate system. Further, the mapping unit 111 determines the positions of all the depth pixels on the color image based on all the world coordinates, and determines the color pixels at the positions on the color image. Further, the electronic device fuses each depth pixel with each color pixel to obtain the target image.

[0170] Among them, the preset depth coordinate system and the preset world coordinate system can be obtained from an open source system, or can be arbitrarily set by the user according to the application scenario, and this application does not impose any restrictions on this.

[0171] Through the above implementation, a target image including a depth value can be generated, so as to subsequently determine the height of the plant to be detected.

[0172] The detection unit 112 detects the color image using a pre-trained MobileNet-SSD network to obtain a detection frame with the plant to be detected.

[0173] In at least one embodiment of the present application, the frame to be detected is obtained by performing feature extraction on the color image using a convolution kernel in the mobilenet-ssd network.

[0174] In at least one embodiment of the present application, before using a pre-trained Mobilenet-SSD network to detect the color image and obtain a detection frame with the plant to be detected, the determination unit 114 determines the field to which the plant to be detected belongs. Further, the acquisition unit 110 acquires multiple color training images on the field. Further, the fusion unit 115 fuses the acquired multiple color training images to obtain a data set. The division unit 116 divides the data set into a training set and a validation set. Further, the training unit 117 uses the color training images in the training set to train an open source convolutional network to obtain a learner. Further, the adjustment unit 118 uses the color training images in the validation set to adjust the learner to obtain the Mobilenet-SSD network.

[0175] In at least one embodiment of the present application, before dividing the data set into a training set and a validation set, the calculation unit 119 calculates the number of color training images in the data set. When the number is less than a preset number, the enhancement unit 120 uses a data enhancement algorithm to increase the number of color training images in the data set.

[0176] Through the above implementation, it is possible to avoid the poor generalization ability of the trained Mobilenet-SSD network due to insufficient number of color training images.

[0177] In at least one embodiment of the present application, the dividing unit 116 divides the data set into a training set and a validation set including:

[0178] The division unit 116 randomly divides the data set into at least one data packet according to a preset ratio, determines any one of the at least one data packet as the verification set, and determines the remaining data packets as the training set, and repeats the above steps until all the data packets are used as the verification set in turn.

[0179] Among them, the preset ratio can be customized and is not limited in this application.

[0180] By dividing the data set through the above implementation method, each color training image in the data set participates in training and verification, thereby improving the fitness of the training of the mobilenet-ssd network.

[0181] In at least one embodiment of the present application, the adjusting unit 118 adjusts the learner using the color training images in the validation set, and obtaining the mobilenet-ssd network includes:

[0182] The adjustment unit 118 uses a hyperparameter grid search method to determine the optimal hyperparameter point from the verification set. Further, the adjustment unit 118 adjusts the learner through the optimal hyperparameter point to obtain the mobilenet-ssd network.

[0183] Specifically, the adjustment unit 118 splits the verification set according to a fixed step size to obtain a target subset, traverses the parameters of the two end points on the target subset, verifies the learner through the parameters of the two end points, obtains the learning rate of each parameter, determines the parameter with the best learning rate as the first hyperparameter point, and within the neighborhood of the first hyperparameter point, reduces the step size and continues to traverse until the step size is a preset step size, that is, the obtained hyperparameter point is the optimal hyperparameter point. Furthermore, the adjustment unit 118 adjusts the learner according to the optimal hyperparameter point to obtain the mobilenet-ssd network.

[0184] Among them, this application does not limit the preset step size.

[0185] Through the above implementation, the mobilenet-ssd network can be made more suitable for the detection of color images in the field.

[0186] In at least one embodiment of the present application, the detection unit 112 detects the color image using a pre-trained mobilenet-ssd network, and obtains a detection frame with the plant to be detected, including:

[0187] The detection unit 112 obtains the deep convolution kernel and the point convolution kernel in the mobilenet-ssd network. Further, the detection unit 112 uses the deep convolution kernel to extract the features of the color image to obtain a first feature map. Further, the detection unit 112 uses the point convolution kernel to process the first feature map to obtain the detection frame.

[0188] Among them, the depth convolution kernel can be a 16*16*128 matrix, and further, the point convolution kernel can be a 1*1*16 matrix.

[0189] The detection frame can be quickly detected through the pre-trained mobilenet-ssd network, thereby improving the detection efficiency.

[0190] The extraction unit 113 extracts the target outline of the plant to be detected from the detection frame.

[0191] In at least one embodiment of the present application, the target contour refers to a contour after irrelevant information is removed from the detection frame, and the shape of the target contour is determined according to the shape of the plant to be detected.

[0192] In at least one embodiment of the present application, the extracting unit 113 extracts the target contour of the plant to be detected from the detection frame, including:

[0193] The extraction unit 113 deletes the background image in the detection frame to obtain a grayscale image. Further, the extraction unit 113 detects the contour of the plant to be detected on the grayscale image to obtain the target contour.

[0194] By deleting the background image, the interference caused by the background image can be avoided, thereby improving the accuracy of extracting the target contour.

[0195] The determining unit 114 determines depth values ​​of all pixels in the target contour according to the target image.

[0196] In at least one embodiment of the present application, the depth value refers to the height of a feature point on the plant to be detected corresponding to a pixel from a camera device.

[0197] In at least one embodiment of the present application, the determining unit 114 determines the depth values ​​of all pixels in the target contour according to the target image, including:

[0198] The determining unit 114 determines a target position of each pixel point on the target image. Further, the determining unit 114 obtains a depth value at the target position from the target image as a depth value of each pixel point.

[0199] By determining the depth values ​​of all the pixels in the target contour according to the target image, the depth values ​​of all the pixels can be determined accurately and quickly.

[0200] The determining unit 114 performs denoising on all the depth values ​​to obtain a target depth value, and determines the height of the plant to be detected according to the target depth value.

[0201] In at least one embodiment of the present application, the determining unit 114 performs denoising on all the depth values ​​to obtain a target depth value, and determines the height of the plant to be detected according to the target depth value, including:

[0202] The determination unit 114 obtains a depth value equal to a preset value from all the depth values, and determines the obtained depth value as a zero value. Further, the determination unit 114 deletes the zero value from all the depth values, and determines the remaining depth value as the target depth value. Further, the determination unit 114 determines the number of the target depth values. The determination unit 114 calculates the sum of the target depth values ​​and divides the sum by the number to obtain the distance between the plant to be detected and the camera device. Further, the determination unit 114 determines the camera height of the location of the camera device, and the determination unit 114 subtracts the camera height from the distance to obtain the height of the plant to be detected.

[0203] By performing denoising on all the depth values ​​to obtain a target depth value, it is possible to again ensure that the target depth value does not contain a depth value with irrelevant information, and the height of the plant to be detected can be accurately determined based on the target depth value.

[0204] In at least one embodiment of the present application, after determining the height of the plant to be detected according to the target depth value, when the height is less than a preset height, the generation unit 121 generates an alarm message according to the height, further, the encryption unit 122 encrypts the alarm message using a symmetric encryption algorithm to obtain a ciphertext, further, the determination unit 114 determines an alarm level of the ciphertext according to the plant to be detected, the determination unit 114 determines an alarm mode according to the alarm level, further, the sending unit 123 sends the ciphertext in the alarm mode.

[0205] Among them, the preset height can be set according to the expected growth rate of the plant to be detected, and the present application does not limit the value of the preset height.

[0206] Furthermore, the alarm levels include: level one, level two, etc.

[0207] Furthermore, the alarm method includes: an alarm sound from a speaker, an email, a telephone call, etc.

[0208] Through the above implementation, an alarm message can be issued when the height is less than the preset height. In addition, by encrypting the alarm message, it is possible to prevent the alarm message from being tampered with, thereby improving the security of the alarm message. At the same time, the alarm mode is determined according to the alarm level, and the alarm message can be sent in an appropriate alarm mode, making the sending of the alarm message more humane.

[0209] It can be seen from the above technical scheme that the present application can obtain a target image containing a depth value by mapping the color image and the depth image, and then the detection frame can be quickly detected through a pre-trained Mobilenet-SSD network to improve the detection efficiency, and the target contour of the plant to be detected is extracted from the detection frame, and the target contour excluding irrelevant information can be extracted. By determining the depth values ​​of all pixels in the target contour according to the target image, the depth values ​​of all pixels can be accurately and quickly determined, and then by denoising all the depth values ​​to obtain the target depth value, it can be ensured again that there is no depth value containing irrelevant information in the target depth value, and the height of the plant to be detected can be accurately determined according to the target depth value.

[0210] like Figure 4 The figure is a schematic diagram of the structure of an electronic device of a preferred embodiment of the plant height determination method of the present application.

[0211] In one embodiment of the present application, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and a computer program stored in the memory 12 and executable on the processor 13, such as a plant height determination program.

[0212] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the electronic device 1 may also include input and output devices, network access devices, buses, etc.

[0213] The processor 13 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 13 is the computing core and control center of the electronic device 1, and uses various interfaces and lines to connect various parts of the entire electronic device 1, and executes the operating system of the electronic device 1 and various installed applications, program codes, etc.

[0214] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned various plant height determination method embodiments, for example Figure 2 Steps shown.

[0215] Exemplarily, the computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an acquisition unit 110, a mapping unit 111, a detection unit 112, an extraction unit 113, a determination unit 114, a fusion unit 115, a division unit 116, a training unit 117, an adjustment unit 118, a calculation unit 119, an enhancement unit 120, a generation unit 121, an encryption unit 122, and a sending unit 123.

[0216] The memory 12 can be used to store the computer program and / or module, and the processor 13 realizes various functions of the electronic device 1 by running or executing the computer program and / or module stored in the memory 12, and calling the data stored in the memory 12. The memory 12 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 12 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0217] The memory 12 may be an external memory and / or an internal memory of the electronic device 1. Furthermore, the memory 12 may be a memory in a physical form, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0218] If the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0219] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0220] Combination Figure 2The memory 12 in the electronic device 1 stores a plurality of instructions to implement a method for determining the height of a plant, and the processor 13 can execute the plurality of instructions to implement: obtaining a color image and a depth image of the plant to be detected; mapping the color image and the depth image to obtain a target image; detecting the color image using a pre-trained Mobilenet-SSD network to obtain a detection frame with the plant to be detected; extracting the target contour of the plant to be detected from the detection frame; determining the depth values ​​of all pixels in the target contour according to the target image; denoising all the depth values ​​to obtain a target depth value, and determining the height of the plant to be detected according to the target depth value.

[0221] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0222] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0223] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0224] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0225] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present application is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0226] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in this application can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0227] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A method for determining plant height, characterized in that: The plant height determination method comprises: Acquire a color image and a depth image of the plant to be detected; Mapping the color image and the depth image to obtain a target image; Using a pre-trained mobilenet-ssd network to detect the color image, and obtain a detection frame with the plant to be detected; Extracting the target outline of the plant to be detected from the detection frame; Determine the depth values ​​of all pixels in the target contour according to the target image; Denoising is performed on the depth values ​​of all the pixels to obtain a target depth value, and the height of the plant to be detected is determined according to the target depth value.

2. The method for determining plant height according to claim 1, wherein: The method of obtaining the color image and the depth image of the plant to be detected includes one or more of the following methods: Using a first lens of a camera device to photograph the plant to be detected to obtain the color image, and using a second lens of the camera device to photograph the plant to be detected to obtain the depth image; and / or The label of the plant to be detected is determined, and an image with the label is obtained from a first configuration library as the color image, and an image with the label is obtained from a second configuration library as the depth image.

3. The method for determining plant height according to claim 2, wherein: The denoising process is performed on the depth values ​​of all the pixels to obtain a target depth value, and determining the height of the plant to be detected according to the target depth value comprises: Acquire a depth value equal to a preset value from the depth values ​​of all the pixels, and determine the acquired depth value as a zero value; Deleting the zero value from all the depth values ​​and determining the remaining depth value as the target depth value; determining the number of target depth values; Calculating the sum of the target depth values; Dividing the sum by the number to obtain the distance between the plant to be detected and the camera device; Determining the camera height of the camera device; The camera height is subtracted from the distance to obtain the height of the plant to be detected.

4. The method for determining plant height according to claim 1, wherein: The mapping process of the color image and the depth image to obtain a target image comprises: Obtaining all depth pixels on the depth image; Mapping all the depth pixels into a preset depth coordinate system to obtain depth coordinates of all the depth pixels; Determine the world coordinates of all the depth pixels according to all the depth coordinates and a preset world coordinate system; Determine the positions of all the depth pixels on the color image according to all the world coordinates, and determine the color pixels at the positions on the color image; Each depth pixel is fused with each color pixel to obtain the target image.

5. The method for determining plant height according to claim 1, wherein: The method of detecting the color image using a pre-trained mobilenet-ssd network to obtain a detection frame with the plant to be detected includes: Obtain the depth convolution kernel and the point convolution kernel in the mobilenet-ssd network; Extracting features of the color image using the deep convolution kernel to obtain a first feature map; The first feature map is processed by using the point convolution kernel to obtain the detection frame.

6. The method for determining plant height according to claim 1, wherein: The step of extracting the target contour of the plant to be detected from the detection frame comprises: Deleting the background image in the detection frame to obtain a grayscale image; The outline of the plant to be detected is detected on the grayscale image to obtain the target outline.

7. The method for determining plant height according to claim 1, wherein: Determining the depth values ​​of all pixels in the target contour according to the target image comprises: Determine the target position of each pixel on the target image; A depth value at the target position is obtained from the target image as the depth value of each pixel.

8. A plant height determination device, characterized in that: The plant height determination device comprises: An acquisition unit, used for acquiring a color image and a depth image of the plant to be detected; A mapping unit, used for mapping the color image and the depth image to obtain a target image; A detection unit, used to detect the color image using a pre-trained mobilenet-ssd network to obtain a detection frame with the plant to be detected; An extraction unit, used for extracting the target outline of the plant to be detected from the detection frame; A determination unit, configured to determine depth values ​​of all pixels in the target contour according to the target image; The determination unit is further configured to perform denoising on the depth values ​​of all the pixels to obtain a target depth value, and determine the height of the plant to be detected according to the target depth value.

9. An electronic device, characterized in that: The electronic device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the plant height determination method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the plant height determination method as described in any one of claims 1 to 7.

Images