Crop female ear spinning identification method and device, electronic equipment and storage medium

Abstract

The invention discloses a crop female ear spinning identification method and device, electronic equipment and a storage medium. The method is executed by a central processing control unit on the robot, and the robot is further provided with a panoramic image acquisition unit. The method comprises the following steps: acquiring an original panoramic image of field crops acquired by the panoramic image acquisition unit; determining a plant interested image according to the original panoramic image; performing female ear positioning on the plant interested image to generate a female ear bounding box, performing female ear coarse classification on an image region corresponding to the female ear bounding box, and outputting a coarse classification result of whether the image region is a female ear; and if the rough classification result is a female ear, performing female ear spinning fine classification on an image area corresponding to the female ear bounding box to obtain a spinning state of a target female ear.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method and device for identifying silking of female ears of crops, an electronic device, and a storage medium. BACKGROUND

[0002] Corn is a major food crop worldwide, and its yield monitoring and breeding screening is extremely important. The silking stage of female ears (R1 stage) is a key growth stage that determines the pollination and fruiting rate and the final yield. Accurate assessment of the silking state of female ears at this stage is one of the core tasks for guiding field management and variety selection.

[0003] Currently, there are defects in the industry and academia in the identification of silking of female ears of crops. The manual field investigation scheme is a labor-intensive operation, which is low in efficiency and high in cost, and cannot meet the high-throughput monitoring demand. In addition, it is highly subjective and has poor data consistency, and also causes physical damage to crops. The aerial remote sensing scheme based on unmanned aerial vehicles has a fundamental information blocking problem due to the "overhead" perspective, and cannot accurately identify and assess the silking state of female ears. The near-ground remote sensing scheme based on large ground platforms has poor accessibility and causes serious damage to crops. The limited field angle of view of the sensor leads to incomplete data collection and low efficiency, and may also cause data omission due to the control blind area. SUMMARY

[0004] The present application provides a method and device for identifying silking of female ears of crops, an electronic device, and a storage medium to improve the accuracy and efficiency of identifying silking of female ears of crops.

[0005] In a first aspect, the embodiments of the present application provide a method for identifying silking of female ears of crops, which is executed by a central processing control unit on a robot, characterized in that the robot also carries a panoramic image acquisition unit; the method comprises:

[0006] acquiring an original panoramic image of field crops collected by the panoramic image acquisition unit;

[0007] determining a plant image of interest according to the original panoramic image;

[0008] locating female ears in the plant image of interest to generate a female ear bounding box, and performing coarse classification of female ears on an image region corresponding to the female ear bounding box to output a coarse classification result of whether it is a female ear;

[0009] if the coarse classification result is a female ear, performing fine classification of female ear silking on the image region corresponding to the female ear bounding box to obtain a silking state of a target female ear.

[0010] In a second aspect, the embodiments of the present application further provide a device for identifying female ear and silk shedding of crops, which is configured to a central processing control unit on a robot, and a panoramic image acquisition unit is also carried on the robot; the device comprises:

[0011] a panoramic image acquisition module, configured to acquire an original panoramic image of crops in a field collected by the panoramic image acquisition unit;

[0012] an image of interest module, configured to determine a plant image of interest according to the original panoramic image;

[0013] a female ear positioning and coarse classification module, configured to perform female ear positioning on the plant image of interest to generate a female ear bounding box, and perform female ear coarse classification on an image region corresponding to the female ear bounding box to output a coarse classification result of whether it is a female ear;

[0014] a silk shedding state classification module, configured to, if the coarse classification result is a female ear, perform female ear silk shedding fine classification on the image region corresponding to the female ear bounding box to obtain a silk shedding state of a target female ear.

[0015] In a third aspect, the embodiments of the present application further provide an electronic device, which comprises:

[0016] one or more processors;

[0017] a storage device, configured to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the methods for identifying female ear and silk shedding of crops provided by the embodiments of the present application.

[0019] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any one of the methods for identifying female ear and silk shedding of crops provided by the embodiments of the present application.

[0020] In a fifth aspect, the embodiments of the present application further provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement any one of the methods for identifying female ear and silk shedding of crops provided by the embodiments of the present application.

[0021] The technical scheme of the present application acquires an original panoramic image through a robot carrying a panoramic image acquisition unit, extracts a plant image of interest from the original panoramic image, separates the target positioning and classification tasks at the feature layer through a hierarchical multi-task decoupling detection head, adopts independent detection branches to avoid feature competition between tasks, and constructs the classification task as a hierarchical structure: the first layer performs a coarse-grained "whether it is a female ear" binary classification, and the second layer performs a fine-grained multi-classification of the spinning state based on the female ear confirmation through a fine-grained detection head, which can focus on learning the tiny visual feature differences of different spinning states, thereby improving the accuracy and efficiency of spinning state recognition. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1a is a flowchart of a crop female ear spinning recognition method according to an embodiment of the present application;

[0023] Figure 1b is a structural schematic diagram of a robot according to an embodiment of the present application;

[0024] Figure 2 is a flowchart of another crop female ear spinning recognition method according to an embodiment of the present application;

[0025] Figure 3 is a structural schematic diagram of a crop female ear spinning recognition device according to an embodiment of the present application;

[0026] Figure 4 is a structural schematic diagram of an electronic device for implementing a crop female ear spinning recognition method according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and above-described accompanying drawings are intended to distinguish similar objects and not necessarily describe a particular chronological or sequential order. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment one

[0030] Figure 1a is a flow chart of a crop ear silking recognition method provided according to Embodiment one of the present application. The present embodiment can be applied to the case of recognizing the ear silking state of crops, and can be executed by a crop ear silking recognition device. The crop ear silking recognition device can be realized in the form of hardware and / or software, and can be configured in a central processing control unit on a robot. The robot also carries a panoramic image acquisition unit. In combination with Figure 1a and Figure 1b , the method comprises:

[0031] S101, acquiring an original panoramic image of field crops collected by the panoramic image acquisition unit;

[0032] S102, determining a plant image of interest according to the original panoramic image;

[0033] S103, performing ear positioning on the plant image of interest to generate an ear bounding box, and performing ear rough classification on the image region corresponding to the ear bounding box to output a rough classification result of whether it is an ear;

[0034] S104, if the rough classification result is an ear, performing ear silking fine classification on the image region corresponding to the ear bounding box to obtain the silking state of the target ear.

[0035] Reference Figure 1bThe robot is provided with a central processing control unit and a panoramic image acquisition unit. The central processing control unit can be an embedded high-performance computing device integrated into the robot platform, serving as the "brain" of the robot. It is responsible for running navigation algorithms, controlling platform movement, triggering camera shooting, processing collected image data in real time or batch, and executing corresponding ear identification algorithms. The panoramic image acquisition unit can be integrated with two back-to-back 220° fisheye cameras, which can respectively acquire fisheye images and stitch them into spherical panoramic images. The panoramic image acquisition unit has one-time imaging capability, which can completely capture the surrounding information of all plants on both sides of the row, and solves the problems of narrow field of view of traditional cameras, the need for complex scanning mechanism (which is easy to cause data blind area) and the possibility of data omission. Due to the absence of mechanical scanning, the system structure is greatly simplified, the failure rate is reduced, and the efficiency and integrity of data acquisition are significantly improved. This non-destructive feature ensures the natural state of crop growth, making continuous and repeated phenotype monitoring data more authentic and reliable, especially suitable for long-term observation of valuable breeding materials.

[0036] The robot travels at a set speed (such as 0.2 m / s) in the row of crops (such as corn, sorghum) according to a preset path or autonomous navigation. During the travel, the panoramic image acquisition unit carried on the robot triggers shooting at fixed time intervals, such as 30HZ, continuously acquiring original panoramic images containing crops on both sides of the row. The robot can also carry a navigation positioning unit to record the accurate geographic coordinates associated with each original panoramic image.

[0037] The central processing control unit obtains the spherical original panoramic image collected by the global image acquisition unit, projects the spherical original panoramic image into a two-dimensional global image, extracts the region of interest (ROI) from the two-dimensional global image, obtains the plant region of interest, removes the invalid region outside the plant, and improves the efficiency of subsequent ear identification processing.

[0038] In an optional embodiment, the plant region of interest image is determined according to the original panoramic image, including: using an equirectangular projection algorithm to expand the original panoramic image into a rectangular panoramic image; based on color threshold and edge detection, the sky area is segmented and removed from the rectangular panoramic image; the ground level line is detected by Hough transform, and the ground area is removed according to the preset canopy height; the rectangular crop is cut along the detected upper and lower boundaries of the canopy to obtain the plant region of interest image including the crop canopy.

[0039] Exemplarily, the original spherical panoramic image is unfolded into a rectangular panoramic image by using an equirectangular projection algorithm, and the spatial geometric information of the spherical panoramic image is completely preserved by preserving the mapping relationship between the spherical coordinates and the plane coordinates. Based on color threshold segmentation (for example, the brightness component V>0.8 in the HSV space) and edge detection, such as Canny operator double-threshold detection, with 30 as the low threshold and 100 as the high threshold, the sky area is accurately segmented and removed from the rectangular panoramic image, and the segmentation accuracy of the sky area is improved through double-modal feature fusion. The ground level line is detected by Hough transform, and the ground area is determined and removed in combination with a preset canopy height threshold (for example, the bottom of the plant canopy is 0.5-1.2 m away from the ground, which is obtained by radar point cloud calibration), and the canopy height constraint is used to avoid false deletion of the bottom of the canopy. Rectangular cropping is performed along the detected upper and lower boundaries of the canopy to obtain an effective area including the crop plant canopy as a plant image of interest, so as to improve the adaptation of the plant image of interest to the actual height of the canopy, significantly improve the accuracy of the plant image of interest, and thus reduce the computational amount of subsequent ear identification.

[0040] Exemplarily, the plant image of interest is subjected to ear regression positioning and ear coarse classification respectively, and an ear bounding box, i.e., ear center coordinates, width, height and confidence, and a coarse classification result, such as whether the ear is greater than a preset coarse-grained confidence threshold, for example, 0.85, are output. The target positioning (regression bounding box) and classification tasks are separated at the feature layer by a hierarchical multi-task decoupled head, an independent detection branch is used to avoid competition between the tasks, and the classification task is constructed as a hierarchical structure: the first layer performs coarse-grained "whether it is an ear" binary classification, and when the confidence is greater than or equal to the coarse-grained confidence threshold, it is determined to be an ear; the second layer, on the basis of the ear confirmation, performs fine multi-classification (no spinning / early spinning / peak spinning / late spinning) of the spinning state by adding an attention mechanism to the fine-grained detection head, and the hierarchical decoupling strategy can focus on learning the tiny visual feature differences (such as silk length, color gradient) of different spinning states, thereby improving the recognition accuracy of the spinning state. If the ear coarse classification result is not an ear, the subsequent fine classification process is directly terminated, and this early termination mechanism can reduce the processing time of a single image and significantly improve the efficiency of real-time processing in the field.

[0041] The technical scheme of the embodiment is characterized in that a robot carrying a panoramic image acquisition unit is used to acquire an original panoramic image, and a plant image of interest is extracted from the original panoramic image; a layered multi-task decoupling detection head is used to separate the target positioning and classification tasks at the feature layer; an independent detection branch is used to avoid feature competition between tasks; and the classification task is constructed as a layered structure: the first layer performs a coarse-grained "whether it is a female ear" binary classification, and the second layer performs a fine-grained multi-classification of the spinning state on the basis of female ear confirmation, so that the detection head can focus on learning the tiny visual feature differences of different spinning states, thereby improving the accuracy and efficiency of spinning state recognition.

[0042] In an optional embodiment, the robot further comprises a mobile platform and a liftable sensor mast; the mobile platform adopts a tracked chassis, and the overall width of the robot is less than a preset crop planting row distance value, so that the mobile platform can pass under the canopy of crops after the canopy is closed; the mobile platform is configured with a motor driving system powered by a lithium battery; the liftable sensor mast is vertically installed on the mobile platform, and the global image acquisition unit is carried on the liftable sensor; the height of the liftable sensor is adjusted according to the actual height of the female ear of crops, so that the global image acquisition unit collects images at a perspective angle parallel to the side of the female ear.

[0043] The robot body comprises a mobile platform, a liftable sensor mast, a panoramic image acquisition unit, and a central processing control unit. The mobile platform adopts a tracked chassis, and the overall width of the robot is set to be less than the conventional crop planting row distance (for example, the width is 40-50 cm), forming a narrow mobile platform, which ensures that the mobile platform can pass under the canopy of crops after the canopy is closed. The mobile platform is configured with a motor driving system powered by a lithium battery, which takes into account environmental protection and continuous operation capability, and is suitable for complex terrains in the field.

[0044] The liftable sensor mast is vertically installed on the mobile platform, and the panoramic image acquisition unit is carried on the top of the mast. The height of the mast can be flexibly adjusted, and the adjustment range is 20-114 cm, which supports manual or automatic mode adjustment. In the manual mode, the height is adjusted manually, so that the panoramic image acquisition unit collects images at a perspective angle parallel to the side of the female ear. In the automatic mode, the actual height of the female ear in the monitoring area can be obtained through a crop phenotype database (which stores the standard height of female ears of different varieties and growth stages) or a laser radar ranging module, and the mast is driven to adjust, so that the center line of the lens of the global image acquisition unit is always aligned with the female ear observation area, thereby improving the image acquisition quality.

[0045] The embodiment of the present application discards the "overlooking" remote sensing of the traditional unmanned aerial vehicle and the "witnessing" remote sensing of the large platform, and initiatively adopts an immersive mode of "going deep into the inside of the canopy and observing along the way". Through a specially designed narrow inter-row robot platform, the sensor can enter the inside of the completely closed crop canopy without damage, and can observe in a nearly parallel side view angle with the ear. The platform width is smaller than the conventional crop planting row distance, and can flexibly pass through the inter-row at the time when the canopy is completely closed in the middle and late growth period of the crops, without damaging the stems, leaves and root systems of the plants. The narrow body solves the problem that the large platform cannot enter the narrow row distance field, avoids physical damage to the crops, and overcomes the problems of manual investigation and damage to the crops by the large ground platform. The side view angle going deep into the inside of the canopy physically completely avoids the serious shielding caused by the upper leaves and the male ear, which is a fundamental defect that cannot be overcome by the overlooking scheme of the unmanned aerial vehicle.

[0046] Embodiment two

[0047] Figure 2 It is a flow chart of another crop ear silking identification method provided according to the embodiment two of the present application. The technical scheme of the present embodiment is further refined on the basis of the above technical scheme. Referring to the crop ear silking identification method shown in FIG. 8, the method comprises the following steps. Figure 2

[0048] S201, acquiring an original panoramic image of field crops collected by the panoramic image collection unit;

[0049] S202, determining a plant image of interest according to the original panoramic image;

[0050] S203, inputting the plant image of interest into a feature extraction module of the panoramic ear identification model for multi-scale feature fusion to obtain a plant feature map including ear shape and spatial distribution information;

[0051] S204, inputting the plant feature map into an ear positioning regressor of the panoramic ear identification model to obtain an ear bounding box;

[0052] S205, inputting a feature map corresponding to the ear bounding box into a coarse-grained classification detection head of the panoramic ear identification model to output a coarse classification result of whether it is an ear;

[0053] S206, if the coarse classification result is an ear, inputting the feature map corresponding to the ear bounding box into a fine-grained classification detection head in the panoramic ear identification model to output a silking state of the target ear; the silking state is non-silking, early silking, full silking or late silking.

[0054] ​An example is to construct a large-scale crop ear panoramic image annotation dataset. A large number of sample images are collected, and the ears in the unfolded two-dimensional panoramic images are annotated by agricultural experts. The annotation information includes: target position (the ear is framed by a bounding box) and silking state (for example: "no silking", "early silking", "full silking", "late silking" and other multi-level classification). The initial network model is trained using the annotation dataset to obtain the panoramic ear recognition model.

[0055] The global ear recognition model includes a feature extraction module, an ear positioning regressor, a coarse-grained classification detection head, and a fine-grained classification detection head. The plant image of interest is input into the feature extraction module of the panoramic ear recognition model, multi-scale feature fusion is performed through a convolutional neural network (CNN), and a plant feature map containing ear shape and spatial distribution information is obtained; the plant feature map is input into the ear positioning regressor of the panoramic ear recognition model, the anchor mechanism is used to regress the accurate position of the ear in the image, and the ear bounding box with confidence score is output; according to the position of the ear bounding box, the corresponding feature map is cropped from the plant feature map; the corresponding feature map is input into the coarse-grained classification detection head of the panoramic ear recognition model, and a binary classification judgment is performed based on a support vector machine (SVM) or a full connection layer, and a coarse classification result of whether the current region is an ear is output; if the coarse classification result is an ear, the feature map corresponding to the ear bounding box is input into the fine-grained classification detection head of the panoramic ear recognition model, the key area of the ear is focused through the attention mechanism, and the silking state of the target ear is output. The silking state is specifically no silking, early silking, full silking, or late silking. Through the coarse-grained binary classification judgment of "whether it is an ear", and on the basis of determining that it is an ear, the second layer performs multi-classification judgment of the fine state such as no silking, early silking, full silking, or late silking. This hierarchical decoupling strategy first completes the binary classification screening of "whether it is an ear" through the coarse-grained detection head, and then the fine-grained detection head performs multi-classification judgment of the silking state of the confirmed ear, so that the fine-grained detection head can focus on learning the subtle visual differences (such as silk color gradient and bending degree) between different silking states, while maintaining real-time processing capability, and improving the accuracy of ear silking recognition.

[0056] In an optional implementation, the plant image of interest is input into the feature extraction module of the panoramic ear recognition model to obtain a plant feature map, including: inputting the plant image of interest into a deformable convolution kernel of the feature extraction module to obtain a distortion-adapted feature map; and inputting the distortion-adapted feature map into a scale self-adaptive attention layer of the feature extraction module to obtain a scale-adapted plant feature map.

[0057] In the feature extraction module, a distortion perception deformable convolution layer and a scale-adaptive attention module are introduced into a traditional convolutional neural network. To solve the problem of ear shape distortion caused by projection distortion in panoramic images, the deformable convolution layer is used to replace part of the standard convolution layer in the front-end and middle-end feature extraction stage of the network. Specifically, the plant image of interest is input into the deformable convolution layer of the feature extraction module, and the adaptive offset of the sampling point is learned to generate a distortion-adaptive feature map. This feature enables the model to actively capture the geometric invariance of the ear, whether it is located in the center of the image or in the edge stretching area, and effectively extract its morphological structure information. To solve the problem of target size difference caused by the change of the distance between the robot and the plant during the robot's movement (coexistence of large targets near and small targets far away), a scale-adaptive attention mechanism is introduced between each layer of the feature pyramid network. Specifically, the distortion-adaptive feature map is input into the module, and the importance weight of the feature map of different scales is learned to generate a scale-adaptive plant feature map. This design enables the network to automatically focus on the feature scale with the most information, achieving fine positioning of the ear near the robot and reliable detection of the ear far away from the robot.

[0058] The technical scheme of the embodiment actively adapts to the geometric distortion of the panoramic image by introducing the deformable convolution, effectively overcomes the severe geometric deformation of the edge area after the panoramic image is unfolded, avoids missing detection and false detection caused by target distortion, improves the recognition accuracy in the entire field of view, and solves the problem of target size change by using the scale-adaptive attention module, improves the detection robustness of the model for multi-scale targets, and ensures comprehensive detection by considering both large-size ears near the robot and small-size ears far away from the robot in the direction of travel, reducing the performance degradation caused by size change. The layered multi-task decoupling detection head is used, the model can focus more on learning the subtle texture and color difference of different silking stages, and the classification accuracy is much higher than that of the traditional integrated detection head, providing more reliable data for agricultural research.

[0059] In an optional embodiment, a navigation positioning unit is also mounted on the robot, and the method further comprises: acquiring geographic coordinates associated with the original panoramic image collected by the navigation positioning unit; calculating the silking rate of the ears in the monitoring area and the proportion of plants in different silking states according to the silking state of the target ear extracted from the original panoramic image and the associated geographic coordinates; based on the silking rate and the proportion of silking states in the monitoring area, different markers are used to mark the positions of each plant on the electronic map to generate a visual field ear silking state distribution map with spatial distribution characteristics.

[0060] The robot can also be equipped with a navigation positioning unit. The navigation positioning unit can include a high-precision real-time kinematic differential global positioning system (RTK-GPS) and an inertial measurement unit (IMU) for obtaining centimeter-level geographic position coordinates and attitude information of the robot. In addition, the robot can also be equipped with a laser radar (LiDAR) or a depth camera to realize autonomous navigation, dynamic path planning and real-time obstacle avoidance between crop rows.

[0061] For example, the geographic coordinates collected by the navigation positioning unit (which are recorded synchronously with the original panoramic image) are obtained to construct a "image frame-geographic coordinate" mapping relationship to form a spatiotemporal data pair. The silking state recognition result of the female ear recognized by the fine classification detection head is bound to the geographic coordinates of the corresponding original panoramic image to generate a female ear state data set with spatial coordinates. Based on the female ear state data set with spatial coordinates, the silking rate and the proportion of each silking state are respectively counted in a specified monitoring area (such as a field or a small area); the different silking states are differentially marked by color or shape using geographical information system (GIS) technology, and the differential marking result is spatially registered with the electronic map base map to form a field female ear silking state distribution map with spatial analysis function, so as to reveal the spatial difference characteristics of the silking process. By converting discrete detection points into a spatially continuous distribution map, it provides unprecedented data-driven decision support for breeders to evaluate variety uniformity, for farmers to perform variable rate fertilization or guide zoned assisted pollination, and for building more accurate yield prediction models, significantly improving the scientificity and precision of agricultural management.

[0062] Embodiment Three

[0063] Figure 3 is a structural schematic diagram of a crop female ear silking recognition device provided according to Embodiment Three of the present application. The present embodiment can be applied to the case of recognizing the silking state of a crop. The crop female ear silking recognition device can be realized in the form of hardware and / or software. The device can be configured in a central processing control unit on a robot, and the robot can also be equipped with a panoramic image acquisition unit. Referring to Figure 3 The specific structure of the crop female ear silking recognition device 300 is as follows:

[0064] The panoramic image acquisition module 310 is configured to acquire an original panoramic image of a field crop acquired by the panoramic image acquisition unit.

[0065] The image of interest module 320 is configured to determine a plant image of interest according to the original panoramic image.

[0066] The female ear positioning and coarse classification module 330 is configured to perform female ear positioning on the plant image of interest to generate a female ear bounding box, and perform female ear coarse classification on an image region corresponding to the female ear bounding box to output a coarse classification result of whether the image region is a female ear.

[0067] The silk spinning state classification module 340 is configured to, if the coarse classification result is a female ear, perform female ear silk spinning fine classification on the image region corresponding to the female ear bounding box to obtain a silk spinning state of the target female ear.

[0068] In an optional embodiment, the robot further comprises a mobile platform and a liftable sensor mast; the mobile platform adopts a tracked chassis, and a whole machine width is less than a preset crop planting row distance value, so that the mobile platform can pass through between crop rows below a canopy after the canopy is closed;

[0069] The liftable sensor mast is vertically installed on the mobile platform, and the liftable sensor carries the global image acquisition unit; a height of the liftable sensor is adjusted according to an actual height of a female ear of a crop, so that the global image acquisition unit performs image acquisition at a perspective view parallel to a side of the female ear.

[0070] In an optional embodiment, the female ear positioning and coarse classification module 330 comprises:

[0071] The feature extraction unit is configured to input the plant image of interest into a feature extraction module of a panoramic female ear recognition model to perform multi-scale feature fusion, and obtain a plant feature map comprising female ear morphology and spatial distribution information;

[0072] The female ear positioning unit is configured to input the plant feature map into a female ear positioning regressor of the panoramic female ear recognition model to obtain a female ear bounding box;

[0073] The coarse classification unit is configured to input a feature map corresponding to the female ear bounding box into a coarse-grained classification detection head of the panoramic female ear recognition model to output a coarse classification result of whether the feature map is a female ear;

[0074] In an optional embodiment, the silk spinning state classification module 340 is specifically configured to:

[0075] If the coarse classification result is a female ear, the feature map corresponding to the female ear bounding box is input into a fine-grained classification detection head of the panoramic female ear recognition model to output a silk spinning state of the target female ear; the silk spinning state is non-silk spinning, silk spinning initial stage, silk spinning full stage, or silk spinning late stage.

[0076] In an optional embodiment, the feature extraction unit is specifically configured to:

[0077] input the plant image of interest into a deformable convolution kernel of the feature extraction module to obtain a feature map after distortion adaptation;

[0078] input the feature map after distortion adaptation into a scale self-adaptive attention layer of the feature extraction module to obtain a plant feature map after scale adaptation.

[0079] In an optional implementation, the image of interest module 320 includes:

[0080] a rectangular panorama unit configured to unfold the original panorama image into a rectangular panorama using an equirectangular projection algorithm;

[0081] a sky segmentation unit configured to segment and remove a sky region from the rectangular panorama based on color thresholding and edge detection;

[0082] a ground detection unit configured to detect a ground horizontal line through Hough transform and determine and remove a ground region according to a preset canopy height;

[0083] an image of interest unit configured to perform rectangular cropping along the detected upper and lower boundaries of the canopy to obtain a plant image of interest including a crop plant canopy.

[0084] In an optional implementation, the robot further carries a navigation positioning unit; and the device 300 further includes a visualization module, which is specifically configured to:

[0085] obtain geographical coordinates associated with the original panorama image collected by the navigation positioning unit;

[0086] calculate a silking rate of the female spike and a proportion of plants in different silking states in the monitoring area according to the silking state of the target female spike extracted from the original panorama image and the associated geographical coordinates;

[0087] based on the silking rate and the proportion of plants in different silking states in the monitoring area, mark the positions of the plants on an electronic map using differential markers to generate a visual female spike silking state distribution map with spatial distribution characteristics.

[0088] The device for identifying silking of female spikes of crops provided in the embodiments can execute the method for identifying silking of female spikes of crops provided in any of the embodiments, and has corresponding functional modules and beneficial effects of the method for identifying silking of female spikes of crops.

[0089] According to the embodiments of the present application, the present application further provides an electronic device, a readable storage medium and a computer program product.

[0090] Embodiment Four

[0091] Figure 4This is a schematic diagram of the structure of an electronic device 410 implementing the crop female ear silking identification method according to embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0092] like Figure 4 As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0093] Multiple components in electronic device 410 are connected to I / O interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of displays, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0094] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as the method for identifying the silking of female ears in crops.

[0095] In some embodiments, the crop ear silking recognition method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 418. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 410 via, e.g., ROM 412 and / or communication unit 419. When the computer program is loaded onto RAM 413 and executed by processor 411, one or more steps of the crop ear silking recognition method described above can be performed. Alternatively, in other embodiments, processor 411 can be configured to perform the crop ear silking recognition method by other means, e.g., with the aid of firmware.

[0096] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0097] Computer programs implementing methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable apparatus to produce a machine, such that the computer program, when executed, enables the system to implement the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0098] In the context of this application, a computer readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium can be a machine readable signal medium. More specific examples of a machine readable storage medium will include one or more lines of a program of instructions in a transitory signal form, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0099] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0100] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain networks, and the Internet.

[0101] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0102] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in this application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of this application can be achieved, and this application does not limit herein.

[0103] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for identifying the silking of female ears in crops, executed by a central processing and control unit on a robot, characterized in that, The robot is also equipped with a panoramic image acquisition unit; the method includes: Obtain the original panoramic image of the field crops captured by the panoramic image acquisition unit; The plant image of interest is determined based on the original panoramic image; The female ear is located in the image of interest of the plant to generate a female ear bounding box, and the image region corresponding to the female ear bounding box is coarsely classified as a female ear and output as a coarse classification result of whether it is a female ear. If the coarse classification result is female ear, then the image region corresponding to the female ear boundary box is further classified into female ear silking fine classification to obtain the silking state of the target female ear.

2. The method according to claim 1, characterized in that, The robot also includes a mobile platform and a liftable sensor mast; the mobile platform adopts a tracked chassis, and the overall width of the machine is smaller than the preset crop planting row spacing value, so that the mobile platform can travel under the canopy between crop rows after the canopy is closed; the mobile platform is equipped with a lithium battery-powered motor drive system; The liftable sensor mast is vertically mounted on the mobile platform, and the global image acquisition unit is mounted on the liftable sensor. The height of the liftable sensor is adjusted according to the actual height of the female ear of the crop, so that the global image acquisition unit can acquire images from a side view parallel to the female ear.

3. The method according to claim 1, characterized in that, The female ear is located in the image of interest of the plant to generate a female ear bounding box, and a coarse classification of the image region corresponding to the female ear bounding box is performed to output a coarse classification result indicating whether it is a female ear, including: The plant image of interest is input into the feature extraction module of the panoramic female ear recognition model for multi-scale feature fusion to obtain a plant feature map including female ear morphology and spatial distribution information. The plant feature map is input into the female ear localization regression unit of the panoramic female ear recognition model to obtain the female ear bounding box. Input the feature map corresponding to the female ear bounding box into the coarse-grained classification detection head of the panoramic female ear recognition model, and output the coarse classification result of whether it is a female ear; If the coarse classification result is a female ear, then a fine classification of the silking state of the female ear is performed on the image region corresponding to the female ear's bounding box to obtain the silking state of the target female ear, including: If the coarse classification result is a female ear, then the feature map corresponding to the female ear bounding box is input into the fine-grained classification detection head in the panoramic female ear recognition model, and the silking state of the target female ear is output; the silking state is no silking, early silking stage, peak silking stage, or late silking stage.

4. The method according to claim 3, characterized in that, The step of inputting the plant image of interest into the feature extraction module of the panoramic female ear recognition model to obtain the plant feature map includes: The plant image of interest is input into the deformable convolution kernel of the feature extraction module to obtain a feature map after distortion adaptation; The distortion-adapted feature map is input into the scale-adaptive attention layer of the feature extraction module to obtain the scale-adapted plant feature map.

5. The method according to claim 1, characterized in that, The step of determining the plant image of interest based on the original panoramic image includes: An equidistant cylindrical projection algorithm is used to unfold the original panoramic image into a rectangular panoramic image; The sky region is segmented and removed from the rectangular panoramic image based on color thresholding and edge detection. The ground level is detected by Hough transform, and the ground area is determined and removed based on the preset canopy height. Rectangular cropping is performed along the detected upper and lower boundaries of the canopy to obtain plant images of interest that include the crop canopy.

6. The method according to any one of claims 1-5, characterized in that, The robot is also equipped with a navigation and positioning unit; the method further includes: Obtain the geographic coordinates associated with the original panoramic image collected by the navigation and positioning unit; Based on the silking state of the target female ear extracted from the original panoramic image and the associated geographic coordinates, the silking rate of female ears and the proportion of plants with different silking states in the monitoring area are calculated. Based on the silking rate and silking status percentage data within the monitoring area, the location of each plant is marked on an electronic map using differentiated markers, generating a visual field distribution map of female ear silking status with spatial distribution characteristics.

7. A device for identifying crop plants by the silking of female ears, configured on a central processing and control unit of a robot, characterized in that, The robot is also equipped with a panoramic image acquisition unit; the device includes: A panoramic image acquisition module is used to acquire the original panoramic images of field crops collected by the panoramic image acquisition unit; An image of interest module is used to determine the plant image of interest based on the original panoramic image; The female ear localization and coarse classification module is used to locate the female ear in the image of interest of the plant, generate the female ear bounding box, and perform coarse classification of the image region corresponding to the female ear bounding box to output the coarse classification result of whether it is a female ear. The silking state classification module is used to perform fine classification of the silking state of the target female ear in the image region corresponding to the female ear boundary box if the coarse classification result is female ear.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the crop female ear silking identification method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the crop female ear silking identification method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the crop female ear silking identification method according to any one of claims 1-6.

Images