Field Image Processing Method for Agricultural Machinery Operations Based on Image Recognition Technology

Abstract

The present disclosure provides a method for processing field images of agricultural machinery operations based on image recognition technology, which can be applied to the fields of image recognition and agricultural technology. The method includes: obtaining a plurality of first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; performing image recognition on the plurality of first crop test images based on an image recognition model deployed on the agricultural machinery equipment to obtain a plurality of first image recognition results; obtaining a supplementary data set for the field area according to model supplementary parameters obtained after analyzing the plurality of first image recognition results; sending the supplementary data set to the cloud; replacing the original parameters with optimized model parameters received from the cloud to obtain an optimized image recognition model; performing image recognition on a plurality of crop images collected in real time based on the optimized image recognition model to obtain a plurality of second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results.

Description

Technical Field

[0001] The present disclosure relates to the fields of image recognition and agricultural technology, and more particularly to a method for processing field images of agricultural machinery operations based on image recognition technology and agricultural machinery equipment. Background Art

[0002] With the development of visual recognition technology and image recognition technology, it has become particularly important to collect and analyze the growth appearance data of field crops in the field. Specifically, the crop images of field crops can be intelligently analyzed with the help of a model, such as judging the crop growth period, weed growth situation, and crop detection.

[0003] The purpose of field image processing is to make the model usage efficiency consistent with the actual field operation requirements, and to complete the intelligent analysis links that need to be processed in field operations, ultimately achieving the purpose of improving operation efficiency. However, the training of the model often differs from the actual application, mainly reflected in the difference between the picture set of the training model and the actual scenario, and the actual scenario changes with different factors such as soil, climate, light, crop cultivation methods, and crop growth periods, resulting in inaccurate model results. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a method for processing field images of agricultural machinery operations based on image recognition technology and agricultural machinery equipment

[0005] According to a first aspect of the present disclosure, there is provided a method for processing field images of agricultural machinery operations based on image recognition technology, including: obtaining a plurality of first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; performing image recognition on the plurality of first crop test images based on an image recognition model deployed on the agricultural machinery equipment to obtain first image recognition results of the plurality of first crop test images respectively; obtaining a supplementary data set for the field area through the image acquisition unit according to the model supplementary parameters obtained after analyzing the plurality of first image recognition results; sending the supplementary data set to the cloud so that the cloud optimizes the image recognition model according to the supplementary data set; replacing the original parameters of the image recognition model deployed on the agricultural machinery equipment with the optimized model parameters received from the cloud to obtain an optimized image recognition model; and performing image recognition on a plurality of crop images collected in real time based on the optimized image recognition model to obtain a plurality of second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results.

[0006] According to an embodiment of the present disclosure, it further includes: displaying the plurality of first image recognition results to an operator through a display unit of the agricultural machinery equipment; determining label information of the plurality of first image recognition results respectively according to the interaction operation of the operator on the display unit; and determining model supplementary parameters according to the plurality of label information.

[0007] According to an embodiment of the present disclosure, the tag information includes an identified target or a mis-identified target. Based on multiple pieces of tag information, model supplementary parameters are determined, including at least one of the following: determining environmental characteristics of a field area based on the identified targets in the multiple pieces of tag information; determining a first number of model supplementary parameters when the environmental characteristics meet a predetermined condition; determining a second number of model supplementary parameters when the environmental characteristics do not meet the predetermined condition; determining a first number of model supplementary parameters when the number of mis-identified targets in the multiple pieces of tag information meets a predetermined threshold.

[0008] According to an embodiment of the present disclosure, the environmental characteristics include at least one of the following: the type of crops in the field area, the number of identified targets, and the identification accuracy rate of each identified target.

[0009] According to an embodiment of the present disclosure, the model supplementary parameters at least include one of the following: shooting frequency, zoom parameter, zoom method, image resolution, and image ratio.

[0010] According to an embodiment of the present disclosure, based on the model supplementary parameters obtained after analyzing the recognition results of multiple first images, a supplementary data set for the field area is obtained through an image acquisition unit, including: adjusting the image acquisition parameters of the image acquisition unit according to the model supplementary parameters; using multiple second crop test images of a second test area re-acquired based on the image acquisition parameters as the supplementary data set.

[0011] According to an embodiment of the present disclosure, replacing the original parameters of the image recognition model deployed on the agricultural machinery device with the optimized model parameters received from the cloud to obtain an optimized image recognition model, including: replacing the original parameters with the intermediate model parameters received from the cloud to obtain an intermediate image recognition model, where the intermediate model parameters are obtained after the cloud performs an optimization operation on the image recognition model a preset number of times using the supplementary data set; performing image recognition on multiple third crop test images of a third test area based on the intermediate image recognition model to obtain third image recognition results for each of the multiple third crop test images; when the multiple third image recognition results meet the accuracy condition of the agricultural machinery device, using the intermediate model parameters as the optimized model parameters and the intermediate image recognition model as the optimized image recognition model.

[0012] According to an embodiment of the present disclosure, the method further includes: obtaining the recognition speed when the image recognition model performs image recognition; adjusting the operation parameters of the agricultural machinery device when the recognition speed is less than the preset speed, where the preset speed is related to the vehicle speed of the agricultural machinery device.

[0013] According to an embodiment of the present disclosure, the operation parameters include at least one of the following: the vehicle speed of the agricultural machinery equipment, the shooting frequency of the image acquisition unit, and the number of image acquisition units in the working state.

[0014] A second aspect of the present disclosure provides an agricultural machinery equipment, including: an agricultural machinery equipment main body for performing field operations; an image acquisition unit disposed on the agricultural machinery equipment main body for acquiring images; the electronic device includes: one or more processors; a memory for storing one or more computer programs; the one or more processors execute the one or more computer programs to implement the steps of the above-mentioned method for processing field images of agricultural machinery operations based on image recognition technology.

[0015] A third aspect of the present disclosure provides an apparatus for processing field images of agricultural machinery operations based on image recognition technology, including: a first acquisition module for acquiring a plurality of first crop test images of a first test area in a field area through an image acquisition unit of the agricultural machinery equipment; a first recognition module for performing image recognition on the plurality of first crop test images based on an image recognition model deployed on the agricultural machinery equipment to obtain first image recognition results of the plurality of first crop test images respectively; a second acquisition module for acquiring a supplementary data set for the field area through the image acquisition unit according to model supplementary parameters obtained after analyzing the plurality of first image recognition results; a first optimization module for sending the supplementary data set to the cloud so that the cloud optimizes the image recognition model according to the supplementary data set; a second optimization module for replacing the original parameters of the image recognition model deployed on the agricultural machinery equipment with the optimized model parameters received from the cloud to obtain an optimized image recognition model; a second recognition module for performing image recognition on a plurality of crop images collected in real time based on the optimized image recognition model to obtain a plurality of second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results.

[0016] A fourth aspect of the present disclosure provides a computer-readable storage medium, on which a computer program or instruction is stored, and when the computer program or instruction is executed by a processor, the steps of the above-mentioned method for processing field images of agricultural machinery operations based on image recognition technology are implemented.

[0017] A fifth aspect of the present disclosure further provides a computer program product, including a computer program or instruction, and when the computer program or instruction is executed by a processor, the steps of the above-mentioned method for processing field images of agricultural machinery operations based on image recognition technology are implemented.

[0018] Embodiments of the present disclosure deploy an image recognition model on agricultural machinery used for performing field operations. By means of real-time acquisition and real-time optimization, an image recognition model applicable to different field areas can be obtained, and the optimized image recognition model is synchronously used to perform field operations in real time. Thus, embodiments of the present disclosure can, during the field operation process, collect crop images and analyze the images while performing the operation, thereby improving the in-depth utilization of image recognition technology in field operations, improving the automated execution efficiency of field operations, improving the quality of field operations, and meeting the requirements of timeliness and accuracy of field operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Through the following description of the embodiments of the present disclosure with reference to the drawings, the above content and other objects, features, and advantages of the present disclosure will become clearer. In the drawings:

[0020] Figure 1 Schematically shows the system architecture of a method for processing field images of agricultural machinery operations based on image recognition technology according to an embodiment of the present disclosure;

[0021] Figure 2 Schematically shows the flowchart of a method for processing field images of agricultural machinery operations based on image recognition technology according to an embodiment of the present disclosure;

[0022] Figure 3 Schematically shows the application scenario of a method for processing field images of agricultural machinery operations based on image recognition technology according to an embodiment of the present disclosure.

[0023] Figure 4 Schematically shows the block diagram of an electronic device suitable for a method for processing field images of agricultural machinery operations based on image recognition technology according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, it should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0027] In cases where expressions similar to "at least one of A, B, and C, etc." are used, generally, it should be interpreted according to the meaning commonly understood by those of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0028] Embodiments of the present disclosure provide a method for processing field images of agricultural machinery operations based on image recognition technology, including: obtaining a plurality of first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; performing image recognition on the plurality of first crop test images based on an image recognition model deployed on the agricultural machinery equipment to obtain first image recognition results for each of the plurality of first crop test images; obtaining a supplementary data set for the field area through the image acquisition unit according to model supplementary parameters obtained after analyzing the plurality of first image recognition results; sending the supplementary data set to the cloud so that the cloud optimizes the image recognition model according to the supplementary data set; replacing the original parameters of the image recognition model deployed on the agricultural machinery equipment with the optimized model parameters received from the cloud to obtain an optimized image recognition model; and performing image recognition on a plurality of crop images collected in real time based on the optimized image recognition model to obtain a plurality of second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results.

[0029] Figure 1 Schematically shows the system architecture of a method for processing field images of agricultural machinery operations based on image recognition technology according to an embodiment of the present disclosure.

[0030] As Figure 1 shown, the system architecture according to Embodiment 100 may include agricultural machinery equipment 101, a display screen 102 installed in the agricultural machinery equipment 101, a network 103, and a server 104. The network 103 is used to provide a medium for a communication link between the agricultural machinery equipment 101 and the server 104. The network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0031] The user can use the display screen 102 in the agricultural machinery equipment 101 to view the operation status of the image recognition model deployed in the agricultural machinery equipment 101, and can also interact with the server 104 through the display screen 102 in the agricultural machinery equipment 101 to receive or send messages, etc.

[0032] Server 104 may be a server that provides various services, such as training and optimizing the deployment of an image recognition model (for example only) in agricultural machinery device 101.

[0033] It should be noted that the method for processing field images of agricultural machinery operations provided by the embodiments of the present disclosure can generally be executed by agricultural machinery device 101. Correspondingly, the device for processing field images of agricultural machinery operations provided by the embodiments of the present disclosure can generally be arranged in server 104.

[0034] It should be understood that Figure 1 the numbers of agricultural machinery devices, networks, and servers in are merely illustrative. According to actual needs, there can be any number of terminal devices, networks, and servers.

[0035] Figure 2 Schematically shows a flowchart of a method for processing field images of agricultural machinery operations based on image recognition technology according to an embodiment of the present disclosure.

[0036] As Figure 2 shown, this embodiment 200 includes operations S210 to S260.

[0037] In operation S210, multiple first crop test images of a first test area in the field area are acquired through the image acquisition unit of the agricultural machinery device.

[0038] In operation S220, based on the image recognition model deployed in the agricultural machinery device, image recognition is performed on the multiple first crop test images to obtain first image recognition results for each of the multiple first crop test images.

[0039] In operation S230, according to the model supplementary parameters obtained after analyzing the multiple first image recognition results, a supplementary data set for the field area is acquired through the image acquisition unit.

[0040] In operation S240, the supplementary data set is sent to the cloud so that the cloud can optimize the image recognition model according to the supplementary data set.

[0041] In operation S250, the original parameters of the image recognition model deployed in the agricultural machinery device are replaced with the optimized model parameters received from the cloud to obtain an optimized image recognition model.

[0042] In operation S260, based on the optimized image recognition model, image recognition is performed on multiple crop images collected in real time to obtain multiple second image recognition results, so as to control the agricultural machinery device to perform field operations according to the second image recognition results.

[0043] According to an embodiment of the present disclosure, an agricultural machine device refers to a movable vehicle that can carry field operation devices, such as a tractor. Based on the type of the field operation device carried by the agricultural machine device, the agricultural machine device can perform various field operations. For example, an agricultural machine device equipped with a plant protection waterway can perform field crop irrigation operations, field weeding operations, etc., and an agricultural machine device equipped with a seeder can perform field seeding operations, field seedling scattering operations, etc.

[0044] Among them, the field operations performed by the agricultural machine device include field operations based on image recognition technology, that is, the image recognition results based on image recognition technology trigger the agricultural machine device to automatically start, end, and adjust field operations.

[0045] According to an embodiment of the present disclosure, an image acquisition unit is provided on the agricultural machine device. As the agricultural machine device moves, the image acquisition unit can collect crop images at different positions and different regions in the field area in real time. Among them, the image acquisition unit can automatically collect crop images.

[0046] Since the actual scene in the field area changes with different factors such as soil, climate, light, crop cultivation methods, and crop growth periods. For example, the growth of crops is different two days apart, and the growth of crops is different during the day and at night. Therefore, the method of image analysis using crop images pre-collected by devices such as drones does not meet the real-time requirements in the agricultural field. Embodiments of the present disclosure deploy an image recognition model based on image recognition technology on the agricultural machine device, so that the agricultural machine device can perform image analysis on the basis of real-time collected crop images, such as image recognition, and then trigger the execution of field operations in real time through the image recognition results, realizing that the agricultural machine device synchronously collects images and synchronously executes field operations.

[0047] According to an embodiment of the present disclosure, the image recognition model deployed on the agricultural machine device can be pre-trained. However, since different periods, different crops, different lands, etc. will affect the model results, when performing field operations in the field area, it is necessary to collect multiple first crop test images in real time to determine whether the image recognition model deployed on the agricultural machine device needs to be optimized. If optimization is required, it is optimized according to the actual scene of the current field area, further improving the accuracy of field operations for the field area.

[0048] The field area can be a field area divided by a preset land unit, such as one mu of land. The first test area in the field area can be any area in the field area, such as a row of crops.

[0049] When the agricultural machinery is driving in the first test area, multiple first crop test images can be collected by the image acquisition unit, and at the same time, the field operation device can be controlled to perform field operations based on the multiple first crop test images. It should be noted that the multiple first crop test images can not only control the field operation device to perform field operations in the first test area, but also be used to optimize the image recognition model based on the actual scenario of the first test area, improve the quality of field operations in other areas of the field, and there is no need to perform field operations in the first test area again.

[0050] According to an embodiment of the present disclosure, the image recognition model can be constructed based on existing image recognition algorithms, for example, object detection algorithms.

[0051] The multiple first image recognition results can reflect the recognition accuracy and adaptability of the image recognition model to the field area. Therefore, after analyzing the multiple first image recognition results, it can be determined whether the image recognition model needs to be optimized. After analyzing the multiple first image recognition results, if the recognition accuracy of the image recognition model meets a predetermined condition, such as a certain threshold, the image recognition model does not need to be optimized, and the agricultural machinery can be directly controlled to perform field operations in the field area based on the image recognition results of the image recognition model for the field area. If the recognition accuracy of the image recognition model does not meet the predetermined condition, model supplementary parameters are determined so as to obtain a supplementary data set for the field area through the image acquisition unit.

[0052] According to an embodiment of the present disclosure, the supplementary data set is the crop images of the field area, which is used to optimize the image recognition model specifically for the actual scenario of the field area.

[0053] The agricultural machinery can send the supplementary data set to the cloud through network communication connection. The cloud stores the original parameters of the image recognition model deployed on the agricultural machinery. The embodiment of the present disclosure realizes the rapid optimization of the image recognition model with the help of the cloud with relatively large computing power, and then returns the optimized model parameters obtained by the cloud after optimizing the image recognition model to the agricultural machinery, so as to avoid the influence of insufficient computing power of the agricultural machinery on real-time field operations and avoid the cost increase caused by deploying more computing power on the agricultural machinery.

[0054] The agricultural machinery can use the optimized model parameters received from the cloud to replace the original parameters to obtain an optimized image recognition model.

[0055] After obtaining the optimized image recognition model, when the agricultural machinery is driving in the field area, multiple crop images can be collected by the image acquisition unit, and at the same time, the field operation device can be controlled to perform normal field operations based on the multiple crop images.

[0056] Embodiments of the present disclosure deploy an image recognition model on agricultural machinery for performing field operations. By means of real-time acquisition and real-time optimization, an image recognition model applicable to different field areas can be obtained, and the optimized image recognition model is synchronously used to perform field operations in real time. Thus, the embodiments of the present disclosure can, during the field operation process, collect crop images and analyze the images while performing the operation, thereby improving the in-depth utilization of image recognition technology in field operations, improving the automated execution efficiency of field operations, improving the quality of field operations, and meeting the timeliness and accuracy requirements of field operations.

[0057] According to an embodiment of the present disclosure, the method further includes: displaying, to an operator, a plurality of first image recognition results through a display unit of the agricultural machinery; determining label information of each of the plurality of first image recognition results according to an interaction operation of the operator on the display unit; and determining model supplementary parameters according to the plurality of label information.

[0058] According to an embodiment of the present disclosure, the display unit may be a touch-interactive display screen on the agricultural machinery. The operator can view the image recognition results of the image recognition model through the display unit and can also receive prompt information of a warning area with problems. In addition, the operator can also perform operations and view on the display screen, judge the warning area and make timely operation optimization adjustments, or record problems.

[0059] For example, after displaying a plurality of first image recognition results to the operator through the display unit of the agricultural machinery, the operator can view the first image recognition results, such as framed crops, weeds, etc. When the operator views the first image recognition results, the operator can also see whether all weeds and crops in the first crop test image are identified, and then determine the label information of each first image recognition result through interaction with the display screen. The label information includes a label indicating whether the recognition accuracy meets the requirements.

[0060] According to an embodiment of the present disclosure, the label information includes recognition targets or misrecognition targets. Determining model supplementary parameters according to the plurality of label information includes at least one of the following:

[0061] Determining environmental characteristics of the field area according to the recognition targets in the plurality of label information; determining a first number of model supplementary parameters when the environmental characteristics meet a predetermined condition; and determining a second number of model supplementary parameters when the environmental characteristics do not meet the predetermined condition;

[0062] Determining a first number of model supplementary parameters when the number of misrecognition targets in the plurality of label information meets a predetermined threshold.

[0063] According to an embodiment of the present disclosure, the environmental features include at least one of the following: the type of crop in the field area, the number of recognition targets, and the recognition accuracy of each recognition target.

[0064] According to an embodiment of the present disclosure, due to the relatively complex actual scenario in the field area and the large variation over time, the label information determined based on the interactive operation of the operator can reflect the actual environmental features of the current field area.

[0065] In a specific embodiment, when the label information is a recognition target, the environmental features for the entire field area can be determined based on multiple label information. The recognition target can be the crop in the current field area or the weed. For example, according to the recognition target in each label information, the environmental features of the field area are determined, such as the crop growth situation and the weed growth situation.

[0066] When the environmental features meet the predetermined conditions, a first number of model supplementary parameters are determined. When the environmental features do not meet the predetermined conditions, a second number of model supplementary parameters are determined. The first number is less than the second number.

[0067] For example, when the environmental feature is the type of crop in the field area, the environmental features meeting the predetermined conditions mean that the type of crop is the predetermined type of crop. Generally, for a predetermined type of crop, such as rice, etc., it has a special growth environment and is relatively easy for the image recognition model to recognize. Therefore, a relatively small first number of model supplementary parameters can be used for model optimization to reduce the data processing volume; for crops such as sweet potatoes and potatoes, their land environment is relatively complex and it is difficult to distinguish them from weeds, and the accuracy of the image recognition model is easily affected. Therefore, a relatively large second number of model supplementary parameters can be used for model optimization.

[0068] When the environmental feature is the number of recognition targets, the environmental features meeting the predetermined conditions mean that the number of recognition targets is less than the predetermined number threshold. If there are fewer targets to be recognized in the field area, a relatively small first number of model supplementary parameters are used for model optimization; on the contrary, a relatively large second number of model supplementary parameters are used for model optimization.

[0069] When the environmental feature is the recognition accuracy of each recognition target, the environmental features meeting the predetermined conditions mean that the overall recognition accuracy reaches the recognition threshold, or the recognition accuracy of the crop reaches the recognition threshold. If the overall recognition accuracy reaches the recognition threshold, a relatively small first number of model supplementary parameters are used for model optimization; on the contrary, a relatively large second number of model supplementary parameters are used for model optimization.

[0070] Since image recognition may be affected by external factors at individual times, resulting in drastic fluctuations in recognition results, the overall recognition accuracy can be effectively evaluated by calculating the average value of multiple image recognitions.

[0071] In another specific embodiment, in the case where the tag information is a misidentified target, the model supplementary parameters can be determined according to whether the number of misidentified targets meets a predetermined threshold. When the number of misidentified targets meets the predetermined threshold, it indicates that the current environmental features are relatively simple, and a smaller first number of model supplementary parameters can be used.

[0072] According to an embodiment of the present disclosure, the model supplementary parameters at least include one of the following: shooting frequency, zoom parameter, zoom method, image resolution, and image ratio.

[0073] In the above embodiment, the first number or the second number of model supplementary parameters can be randomly determined from the model supplementary parameters.

[0074] In another specific embodiment, a selection page can also be presented to the operator, and the operator can independently select the first number or the second number of model supplementary parameters through the selection page.

[0075] For example, the first number can be 3 and the second number is 5, that is, the second number is all the model supplementary parameters.

[0076] According to an embodiment of the present disclosure, the shooting frequency is the shooting frequency of the image acquisition unit. When determining the model supplementary parameters, it can be adjusted according to a preset shooting frequency or customized according to the interaction of the operator. It should be noted that the embodiments of the present disclosure can perform image recognition and execute field operations in real time through agricultural machinery, and the field operations are triggered by image recognition. If the model operation ability does not match the image acquisition unit, the shooting frequency will be relatively low during high-speed operations, and the image recognition cannot keep up with the vehicle speed, resulting in inaccurate model results. Therefore, both the customized or preset shooting frequencies match the model operation ability of the image recognition model.

[0077] According to an embodiment of the present disclosure, the zoom parameter can be a customized or predetermined focal length, and the zoom method can include manual zoom and automatic zoom.

[0078] The image resolution generally adopts 1920*1280 and 1280*720, and the image ratios are 16:9 and 4:3.

[0079] According to an embodiment of the present disclosure, the shooting frequency, zoom parameter, zoom method, image resolution, and image ratio are all model supplementary parameters for the image acquisition stage, and images matching the actual scene of the current field area can be collected in real time to form a supplementary data set and optimize the image recognition model.

[0080] Specifically, according to the model supplementary parameters obtained after analyzing multiple first image recognition results, a supplementary data set for the field area is obtained through the image acquisition unit, including: adjusting the image acquisition parameters of the image acquisition unit according to the model supplementary parameters; using multiple second crop test images of the second test area re-acquired based on the image acquisition parameters as the supplementary data set.

[0081] For example, through the display screen of the agricultural machinery equipment, the operator can view multiple first image recognition results output by the image recognition model. At the same time, the operator can also view the training set information of the image recognition model, such as parameters of the training set, the type of crops targeted, the setting of the number of recognition targets, and other parameter information. When the recognition rate and accuracy of the first image recognition results are not high, such as less than 80%, it indicates that the generalization of the model is insufficient and more images are needed. Thus, the operator can set parameters such as the shooting frequency, automatic zoom, manual zoom, picture resolution, and picture ratio, and complete the image acquisition of multiple second crop test images in the second test area while the agricultural machinery equipment is performing field operations. After that, the operator can also select a storage directory through the display screen and incorporate the acquired multiple second crop test images into the supplementary data set for model optimization.

[0082] Thus, when optimizing the model, targeted image acquisition can be performed according to each model supplementary parameter, which is convenient for using different model supplementary parameters for image enhancement in the cloud to ensure that the optimized image recognition model can be more applicable to the actual scenario of the current field area.

[0083] The embodiments of the present disclosure increase the supplementary data set by adding model supplementary parameters, enabling the model to have better generalization and improve its adaptability to the current field area. Furthermore, when re-optimizing the model, both the model accuracy and recall rate can be improved.

[0084] In another specific embodiment, the model supplementary parameters may also include model recognition parameters during the model recognition stage.

[0085] Specifically, after image acquisition and image recognition through operation S210 and operation S220, the operator can view the running status of the model and multiple first image recognition results. According to the multiple first image recognition results, the accuracy distribution of the model's image recognition and whether any other targets are misdetected can be seen. At this time, the operator can adjust the model accuracy through interactive operations. For example, if there are more targets to be recognized in the field, the model recognition parameters are increased; conversely, they are decreased. For example, when the accuracy of the recognition frame marking the crops is low during the model operation, the model recognition parameters can be appropriately decreased. This method is suitable for image recognition models with larger targets in the field.

[0086] When adjusting the model recognition parameters, the image recognition model after the model recognition parameters are adjusted can be directly used as the optimized image recognition model.

[0087] According to an embodiment of the present disclosure, the model recognition parameter can be understood as a threshold parameter for judging the type of the recognition target in the output layer of the image recognition model.

[0088] According to an embodiment of the present disclosure, the optimization of the image recognition model by the cloud in operation S240 according to the supplementary data set includes the following four steps.

[0089] Step 1: The cloud randomly classifies the received supplementary data set into a training set and a test set, and classifies all the images in the supplementary data set into two sets. Specifically, the supplementary data set includes supplementary data subsets corresponding to each model supplementary parameter. When classifying, all the images in each supplementary data subset need to be classified into the training set and the test set.

[0090] Step 2: The cloud performs image enhancement processing according to the number and priority of the model supplementary parameters to generate new test images. For example, the image enhancement processing can be operations such as cropping and denoising.

[0091] Step 3: The cloud automatically optimizes the model according to the training loop parameters configured by the user and outputs the optimized model parameters to the agricultural machinery equipment.

[0092] Step 4: If the operator is satisfied with the optimization result, the original parameters of the image recognition model deployed on the agricultural machinery equipment can be replaced with the optimized model parameters received from the cloud to obtain an optimized image recognition model and be used.

[0093] For step 3, a parameter of the number of training rounds can be set, such as training 10 - 30 times. Then, the trained model is continuously verified in the field until the recognition accuracy meets the requirements, and finally the optimized model parameters are output to the agricultural machinery equipment.

[0094] According to an embodiment of the present disclosure, replacing the original parameters of the image recognition model deployed on the agricultural machinery device with the optimized model parameters received from the cloud to obtain an optimized image recognition model includes: replacing the original parameters with the intermediate model parameters received from the cloud to obtain an intermediate image recognition model, where the intermediate model parameters are obtained after the cloud performs an optimization operation on the image recognition model for a preset number of optimization times using a supplementary data set; based on the intermediate image recognition model, performing image recognition on multiple third crop test images in a third test area to obtain third image recognition results for each of the multiple third crop test images; when the multiple third image recognition results meet the accuracy condition of the agricultural machinery device, using the intermediate model parameters as the optimized model parameters and using the intermediate image recognition model as the optimized image recognition model.

[0095] According to an embodiment of the present disclosure, after obtaining the intermediate image recognition model, it is possible to determine whether the intermediate image recognition model is an image recognition model suitable for the current field area based on the third image recognition results for the third test area.

[0096] According to an embodiment of the present disclosure, a verification time parameter can be set, with a minimum of 5 minutes and a maximum of 1 hour. Set the verification time parameter for the training set during optimization and calculate the recognition accuracy within the verification time parameter as a reference for whether the model meets the optimization standard.

[0097] According to an embodiment of the present disclosure, since the computing power consumption of the model on the agricultural machinery device cannot be accurately known before deployment, the types and quantities of image acquisition units deployed on each agricultural machinery device are also different, and the vehicle speeds of each model performing different field operations are also different. Therefore, it is necessary to test the computing power matching between the model and the agricultural machinery device in order to obtain the best usage method of the model.

[0098] Specifically, obtain the recognition speed when the image recognition model performs image recognition; when the recognition speed is less than the preset speed, adjust the operating parameters of the agricultural machinery device, where the preset speed is related to the vehicle speed of the agricultural machinery device.

[0099] In this embodiment, the preset speed can be that when driving 1 meter based on the current vehicle speed, the recognition speed is not less than 10 pictures and not more than 20 pictures. Less than 10 pictures indicates insufficient computing power and does not meet the computing power condition. More than 20 pictures indicates remaining computing power and other adaptive changes can be made.

[0100] After deploying the model on the agricultural machinery device, in the first test area / second test area / third test area, it is possible to synchronously perform tests on image acquisition, image recognition, and field operations, and be able to see the real-time situation of the model running, the marked recognized crop situation, and the current recognition speed of the model, such as XX pictures / mm second.

[0101] Thus, when the recognition speed is less than the preset speed, the operating parameters of the agricultural machinery equipment are adjusted to match the computing power of the model and the agricultural machinery equipment.

[0102] According to an embodiment of the present disclosure, the operating parameters include at least one of the following: the vehicle speed of the agricultural machinery equipment, the shooting frequency of the image acquisition unit, and the number of image acquisition units in the working state.

[0103] For example, when the recognition speed is less than the preset speed, the operator can reduce the vehicle speed, or optimize the model recognition parameters, such as reducing the number of recognition targets, or reducing the number of the image acquisition units in the working state, to reduce the load of each image acquisition hardware. If the speed is higher than 20 pictures, the number of the image acquisition units in the working state can be increased, or the vehicle speed can be increased, or the number of recognition targets can be increased.

[0104] Figure 3 FIG. schematically shows an application scenario of a field image processing method for agricultural machinery operations based on image recognition technology according to an embodiment of the present disclosure.

[0105] As Figure 3 shown, taking the crops in the rightmost column as the first test area 301, after the agricultural machinery equipment 310 passes through the first test area 301, a plurality of first crop test images can be obtained. According to the first image recognition results of the plurality of first crop test images, the model supplementary parameters can be determined. When performing field operations in the second test area 302, a supplementary data set can be obtained according to the model supplementary parameters. The supplementary data set is sent to the cloud 320 for model optimization. After the optimization is completed, the optimized model parameters are received from the cloud, and the original parameters are replaced with the optimized model parameters to obtain an optimized image recognition model deployed on the agricultural machinery equipment. After that, the agricultural machinery equipment 310 can continue to perform field operations on other areas of the field area based on the optimized image recognition model.

[0106] For ease of understanding, the following will describe this solution with an example.

[0107] Fix the image acquisition unit on the agricultural machinery equipment and connect the image acquisition unit to the upper computer of the agricultural machinery equipment for interaction through the display screen of the upper computer. For example, configure and optimize the image recognition model through the display screen. Configuring the image recognition model includes parameter configuration, accuracy detection, cloud training, model selection, model deletion, etc.

[0108] After completing the above configuration, select New Job, Historical Job, or Last Job to complete the job startup configuration. For a new job, you need to configure the name of the current job, select an image recognition model, and after completing the model parameter configuration, select the method for processing the image, such as storing or not storing. After selecting storage, you can choose to record recognition points + the original image or only store the original image. After completing the configuration, you can perform field operations. If you select a historical job, you need to select the name of the historical job and then enter field operations. If you select the last job, the configuration parameters of the most recent job will be used by default.

[0109] After starting the job, the display screen will show the crop images captured by the image acquisition unit. In addition, the image recognition model will perform image recognition on the crop images and display them, as well as label the identified abnormal information. After seeing the abnormal information, the operator can handle it based on experience.

[0110] After completing the detection of the first test area based on the above operations, if the model recognition accuracy is poor, you can adjust the model recognition parameters. If there are more targets to be recognized in the field, increase the model recognition parameters; otherwise, decrease the model recognition parameters.

[0111] When adjusting the model recognition parameters still cannot ensure that the recognition accuracy reaches the application effect, it means that the generalization of the model needs to be improved. Therefore, you can use the device image acquisition function to collect a supplementary dataset based on the model supplementary parameters and upload it to the cloud. After starting the device image acquisition function, the operator can also customize the selection of model supplementary parameters or automatically select the first / second number of model supplementary parameters based on the default options.

[0112] After that, the cloud will automatically optimize the model according to the training loop parameters configured by the user and deploy the optimized model on the agricultural machinery equipment. During this process, you can directly use the optimized model parameters output by the cloud to replace the original parameters to obtain an optimized image recognition model; or use the optimized model parameters to replace the original parameters to obtain an intermediate image recognition model, and determine whether to use the intermediate image recognition model as the optimized image recognition model through actual testing in the third test area.

[0113] Finally, after completing the optimization, you can perform image recognition on multiple real-time captured crop images based on the optimized image recognition model to obtain multiple second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results.

[0114] According to an embodiment of the present disclosure, a field image processing device for agricultural machinery operation based on image recognition technology includes: a first acquisition module, configured to acquire a plurality of first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery; a first recognition module, configured to perform image recognition on the plurality of first crop test images based on an image recognition model deployed on the agricultural machinery to obtain first image recognition results of the plurality of first crop test images respectively; a second acquisition module, configured to acquire a supplementary data set for the field area through the image acquisition unit according to model supplementary parameters obtained after analyzing the plurality of first image recognition results; a first optimization module, configured to send the supplementary data set to the cloud so that the cloud optimizes the image recognition model according to the supplementary data set; a second optimization module, configured to replace original parameters of the image recognition model deployed on the agricultural machinery with optimized model parameters received from the cloud to obtain an optimized image recognition model; a second recognition module, configured to perform image recognition on a plurality of real-time acquired crop images based on the optimized image recognition model to obtain a plurality of second image recognition results, so as to control the agricultural machinery to perform field operations according to the second image recognition results.

[0115] It should be noted that the device part in the embodiment of the present disclosure corresponds to the method part in the embodiment of the present disclosure. For the description of the device part, please refer to the method part specifically, and details are not repeated here.

[0116] According to an embodiment of the present disclosure, any multiple of the first acquisition module, the first recognition module, the second acquisition module, the first optimization module, the second optimization module, and the second recognition module can be combined and implemented in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0117] According to an embodiment of the present disclosure, at least one of the first acquisition module, the first recognition module, the second acquisition module, the first optimization module, the second optimization module, and the second recognition module can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging circuits and other hardware or firmware, or can be implemented in any one of the three implementation manners of software, hardware, and firmware or in any appropriate combination of several of them. Alternatively, at least one of the first acquisition module, the first recognition module, the second acquisition module, the first optimization module, the second optimization module, and the second recognition module can be at least partially implemented as a computer program module, and when the computer program module runs, it can execute corresponding functions.

[0118] Figure 4 A block diagram of an electronic device suitable for a field image processing method of agricultural machinery operation based on image recognition technology according to an embodiment of the present disclosure is schematically shown.

[0119] As Figure 4 shown, the electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage section 408 into a random access memory (RAM) 403. The processor 401 can include, for example, a general microprocessor (e.g., CPU), an instruction set processor, and / or a related chipset and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), etc. The processor 401 can also include on-board memory for caching purposes. The processor 401 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0120] In the RAM 403, various programs and data required for the operation of the electronic device 400 are stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. The processor 401 performs various operations of the method flow according to an embodiment of the present disclosure by executing programs in the ROM 402 and / or the RAM 403. It should be noted that the program can also be stored in one or more memories other than the ROM 402 and the RAM 403. The processor 401 can also perform various operations of the method flow according to an embodiment of the present disclosure by executing programs stored in the one or more memories.

[0121] According to an embodiment of the present disclosure, the electronic device 400 can further include an input / output (I / O) interface 405, and the input / output (I / O) interface 405 is also connected to the bus 404. The electronic device 400 can further include one or more of the following components connected to the input / output I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read from it can be installed into the storage section 408 as needed.

[0122] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the methods according to the embodiments of the present disclosure are implemented.

[0123] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the above-described ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403.

[0124] Embodiments of the present disclosure also include a computer program product, which includes a computer program that contains program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to cause the computer system to implement the above methods provided by the embodiments of the present disclosure.

[0125] When the computer program is executed by the processor 401, the above functions defined in the system / apparatus of the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described systems, apparatuses, modules, units, etc. may be implemented by computer program modules.

[0126] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium and be downloaded and installed through the communication part 409, and / or be installed from the removable medium 411. The program code included in the computer program may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0127] In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above functions defined in the system of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the systems, devices, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0128] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedures and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, such as Java, C++, Python, the "C" language, or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the block may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for executing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0130] Those skilled in the art can understand that the features recited in the various embodiments and / or claims of the present disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly recited in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features recited in the various embodiments and / or claims of the present disclosure can be combined and combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0131] The specific embodiments described above further elaborate on the objectives, technical solutions, and beneficial effects of the present disclosure. It should be understood that the above are only specific embodiments of the present disclosure and are not used to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for processing field images of agricultural machinery operations based on image recognition technology, characterized in that, The method includes: Obtaining a plurality of first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; Performing image recognition on the plurality of first crop test images based on an image recognition model deployed on the agricultural machinery equipment to obtain first image recognition results of the plurality of first crop test images respectively; According to model supplement parameters obtained after analyzing the plurality of first image recognition results, obtaining a supplementary data set for the field area through the image acquisition unit; the model supplement parameters are used to adjust image acquisition parameters of the image acquisition unit, and the model supplement parameters at least include one of the following: shooting frequency, zoom parameter, zoom method, image resolution, image ratio; Sending the supplementary data set to the cloud so that the cloud optimizes the image recognition model according to the supplementary data set; Replacing original parameters of the image recognition model deployed on the agricultural machinery equipment with optimized model parameters received from the cloud to obtain an optimized image recognition model; and Based on the optimized image recognition model, performing image recognition on a plurality of crop images collected in real time to obtain a plurality of second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results; Determining environmental characteristics of the field area according to recognition targets in label information of the plurality of first image recognition results respectively; the environmental characteristics include at least one of the following: crop type in the field area, number of recognition targets, recognition accuracy of each recognition target; Determining a first number of the model supplement parameters when the environmental characteristics meet a predetermined condition; Determining a second number of the model supplement parameters when the environmental characteristics do not meet the predetermined condition; Wherein, the first number is less than the second number, and the environmental characteristics meeting the predetermined condition include at least one of the following: the crop type is a predetermined crop type, the number of recognition targets is less than a predetermined number threshold, the recognition accuracy of the crop reaches a recognition threshold, and the overall recognition accuracy of the plurality of recognition targets reaches the recognition threshold; The method further includes: Displaying the plurality of first image recognition results to an operator through a display unit of the agricultural machinery equipment; Determining label information of the plurality of first image recognition results respectively according to an interactive operation of the operator on the display unit; Determining the model supplement parameters according to the plurality of label information; The obtaining the supplementary data set for the field area through the image acquisition unit according to the model supplement parameters obtained after analyzing the plurality of first image recognition results includes: Adjusting image acquisition parameters of the image acquisition unit according to the model supplement parameters; Taking a plurality of second crop test images of a second test area re-acquired based on the image acquisition parameters as the supplementary data set.

2. The method according to claim 1, wherein The label information includes a recognition target or a mis-recognition target, and the determining the model supplement parameters according to the plurality of label information includes: When the number of mis-identified targets among the multiple pieces of tag information meets a predetermined threshold, determine the first quantity of the model supplementary parameters.

3. The method according to claim 1, characterized in that, The step of using the optimized model parameters received from the cloud to replace the original parameters of the image recognition model deployed on the agricultural machinery device to obtain an optimized image recognition model includes: Using the intermediate model parameters received from the cloud to replace the original parameters to obtain an intermediate image recognition model, where the intermediate model parameters are obtained after the cloud performs an optimization operation on the image recognition model for a preset number of optimization times using the supplementary dataset; Based on the intermediate image recognition model, perform image recognition on multiple third crop test images in a third test area to obtain third image recognition results for each of the multiple third crop test images; When the multiple third image recognition results meet the accuracy condition of the agricultural machinery device, use the intermediate model parameters as the optimized model parameters and use the intermediate image recognition model as the optimized image recognition model.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the recognition speed when the image recognition model performs image recognition; When the recognition speed is less than a preset speed, adjust the operating parameters of the agricultural machinery device, where the preset speed is related to the vehicle speed of the agricultural machinery device.

5. The method according to claim 4, wherein The operating parameters include at least one of the following: the vehicle speed of the agricultural machinery device, the shooting frequency of the image acquisition unit, and the number of image acquisition units in a working state.

6. An agricultural machinery device, comprising: An agricultural machinery device main body for performing field operations; An image acquisition unit provided on the agricultural machinery device main body for acquiring images; An electronic device, including: One or more processors; a memory for storing one or more computer programs; characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 5.

Images