Agricultural machinery operation field image processing method based on image recognition technology

Abstract

The invention provides an agricultural machinery operation field image processing method based on an image recognition technology, which can be applied to the technical fields of image recognition and agriculture. The method comprises the following steps: acquiring a plurality of first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; based on an image recognition model deployed on the agricultural machinery equipment, performing image recognition on the plurality of first crop test images to obtain a plurality of first image recognition results; obtaining a supplementary data set for the field area according to model supplementary parameters obtained after the plurality of first image recognition results are analyzed; sending the supplementary data set to the cloud; replacing the original parameters with optimization model parameters received from the cloud to obtain an optimized image recognition model; and based on the optimized image recognition model, performing image recognition on a plurality of crop images acquired in real time to obtain a plurality of second image recognition results, and controlling the agricultural machinery equipment to execute field operation according to the second image recognition results.

Description

Technical Field

[0001] The present disclosure relates to the fields of image recognition and agricultural technology, and more specifically to a method for processing agricultural machinery operation field images and agricultural machinery equipment based on image recognition technology. Background Art

[0002] With the development of visual recognition technology and image recognition technology, it is particularly important to collect and analyze the growth data of field crops in the field. Specifically, the crop images of field crops can be intelligently analyzed with the help of models, such as judging the crop growth period, weed growth, and crop detection.

[0003] The purpose of field image processing is to achieve consistency between model efficiency and actual field operation requirements, and to complete the intelligent analysis links that need to be processed in field operations, ultimately achieving the goal of improving operation efficiency. However, model training often differs from actual applications, mainly reflected in the difference between the image set used to train the model and the actual scene. The actual scene changes will vary with soil, climate, light, crop cultivation methods, crop growth period, etc., resulting in inaccurate model results. Summary of the invention

[0004] In view of the above problems, the present invention provides a method for processing agricultural machinery field images based on image recognition technology, and an agricultural machinery device.

[0005] According to a first aspect of the present disclosure, a method for processing field images of agricultural machinery operations based on image recognition technology is provided, comprising: acquiring a plurality of first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; performing image recognition on the plurality of first crop test images based on an image recognition model deployed on the agricultural machinery equipment to obtain first image recognition results of each of the plurality of first crop test images; acquiring a supplementary data set for the field area through the image acquisition unit according to model supplementary parameters obtained after analyzing the plurality of first image recognition results; sending the supplementary data set to a cloud so that the cloud optimizes the image recognition model according to the supplementary data set; replacing the original parameters of the image recognition model deployed on the agricultural machinery equipment with the optimized model parameters received from the cloud to obtain an optimized image recognition model; and performing image recognition on a plurality of crop images acquired in real time based on the optimized image recognition model to obtain a plurality of second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results.

[0006] According to an embodiment of the present disclosure, it also includes: displaying multiple first image recognition results to an operator through a display unit of agricultural machinery equipment; determining label information of each of the multiple first image recognition results based on the operator's interactive operation on the display unit; and determining model supplement parameters based on the multiple label information.

[0007] According to an embodiment of the present disclosure, the label information includes an identification target or an incorrectly identified target, and the model supplement parameters are determined based on multiple label information, including at least one of the following: determining environmental characteristics for the field area based on the identification targets in the multiple label information; determining a first number of model supplement parameters when the environmental characteristics meet predetermined conditions; determining a second number of model supplement parameters when the environmental characteristics do not meet the predetermined conditions; determining a first number of model supplement parameters when the number of incorrectly identified targets in the multiple label information meets a predetermined threshold.

[0008] According to an embodiment of the present disclosure, the environmental feature includes at least one of the following: the type of crops in the field area, the number of identified targets, and the recognition accuracy rate of each identified target.

[0009] According to an embodiment of the present disclosure, the model supplementary parameters include at least one of the following: shooting frequency, zoom parameter, zoom mode, image resolution, and image ratio.

[0010] According to an embodiment of the present disclosure, based on the model supplementary parameters obtained after analyzing multiple first image recognition results, a supplementary data set for the field area is obtained through an image acquisition unit, including: adjusting the image acquisition parameters of the image acquisition unit according to the model supplementary parameters; and using multiple second crop test images of the second test area re-acquired based on the image acquisition parameters as the supplementary data set.

[0011] According to an embodiment of the present disclosure, original parameters of an image recognition model deployed on agricultural machinery are replaced with optimized model parameters received from the cloud to obtain an optimized image recognition model, including: replacing original parameters with intermediate model parameters received from the cloud to obtain an intermediate image recognition model, wherein the intermediate model parameters are obtained after the cloud performs a preset number of optimization operations on the image recognition model using a supplementary data set; based on the intermediate image recognition model, performing image recognition on multiple third crop test images of a third test area to obtain third image recognition results of each of the multiple third crop test images; when the multiple third image recognition results meet the accuracy conditions of the agricultural machinery, using the intermediate model parameters as optimized model parameters and using the intermediate image recognition model as the optimized image recognition model.

[0012] According to an embodiment of the present disclosure, the method also includes: obtaining a recognition speed of an image recognition model when performing image recognition; and when the recognition speed is less than a preset speed, adjusting an operating parameter of the agricultural machinery equipment, wherein the preset speed is related to the vehicle speed of the agricultural machinery equipment.

[0013] According to an embodiment of the present disclosure, the operating parameters include at least one of the following: the speed of the agricultural machinery, the shooting frequency of the image acquisition unit, and the number of image acquisition units in working state.

[0014] The second aspect of the present disclosure provides an agricultural machinery device, comprising: an agricultural machinery device main body, used to perform field operations; an image acquisition unit, arranged on the agricultural machinery device main body, used to acquire images; an electronic device comprising: one or more processors; a memory, used to store one or more computer programs; one or more processors execute one or more computer programs to implement the steps of the above-mentioned agricultural machinery operation field image processing method based on image recognition technology.

[0015] The third aspect of the present disclosure provides an agricultural machinery field image processing device based on image recognition technology, including: a first acquisition module, used to acquire multiple first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; a first recognition module, used to perform image recognition on the multiple first crop test images based on an image recognition model deployed on the agricultural machinery equipment, and obtain first image recognition results of each of the multiple first crop test images; a second acquisition module, used to acquire a supplementary data set for the field area through the image acquisition unit according to model supplementary parameters obtained after analyzing the multiple first image recognition results; a first optimization module, used to send the supplementary data set to the cloud, so that the cloud optimizes the image recognition model according to the supplementary data set; a second optimization module, used to replace the original parameters of the image recognition model deployed on the agricultural machinery equipment with the optimized model parameters received from the cloud, and obtain an optimized image recognition model; a second recognition module, used to perform image recognition on multiple crop images collected in real time based on the optimized image recognition model, and obtain multiple second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results.

[0016] A fourth aspect of the present disclosure provides a computer-readable storage medium having a computer program or instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for processing agricultural machinery field images based on image recognition technology.

[0017] The fifth aspect of the present disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of the above-mentioned agricultural machinery field image processing method based on image recognition technology.

[0018] The embodiments of the present disclosure deploy image recognition models on agricultural machinery used to perform field operations, and can obtain image recognition models suitable for different field areas through real-time acquisition and real-time optimization, and simultaneously use the optimized image recognition models to perform field operations in real time. As a result, the embodiments of the present disclosure can realize the acquisition and analysis of crop images while performing field operations, thereby improving the in-depth utilization of image recognition technology in field operations, improving the efficiency of automated execution of field operations, improving the quality of field operations, and meeting the timeliness and accuracy requirements of field operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0020] Figure 1 The system architecture of the method for processing images of agricultural machinery operation fields based on image recognition technology according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 2 The flowchart of the method for processing images of agricultural machinery operation fields based on image recognition technology according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 3 The application scenario of the method for processing images of agricultural machinery operations in the field based on image recognition technology according to an embodiment of the present disclosure is schematically illustrated.

[0023] Figure 4 A block diagram of an electronic device suitable for a method for processing images of agricultural machinery operations in a field based on image recognition technology according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] An embodiment of the present disclosure provides a method for processing field images of agricultural machinery operations based on image recognition technology, comprising: acquiring multiple first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; performing image recognition on the multiple first crop test images based on an image recognition model deployed on the agricultural machinery equipment to obtain first image recognition results of each of the multiple first crop test images; acquiring a supplementary data set for the field area through the image acquisition unit based on model supplementary parameters obtained after analyzing the multiple first image recognition results; sending the supplementary data set to a cloud so that the cloud optimizes the image recognition model based on the supplementary data set; replacing the original parameters of the image recognition model deployed on the agricultural machinery equipment with the optimized model parameters received from the cloud to obtain an optimized image recognition model; and performing image recognition on multiple crop images acquired in real time based on the optimized image recognition model to obtain multiple second image recognition results, so as to control the agricultural machinery equipment to perform field operations based on the second image recognition results.

[0029] Figure 1 The system architecture of the method for processing images of agricultural machinery operations in the field based on image recognition technology according to an embodiment of the present disclosure is schematically shown.

[0030] like Figure 1 As shown, the system architecture according to the embodiment 100 may include agricultural machinery 101, a display screen 102 installed in the agricultural machinery 101, a network 103 and a server 104. The network 103 is used to provide a medium for a communication link between the agricultural machinery 101 and the server 104. The network 103 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0031] The user can use the display screen 102 in the agricultural machinery 101 to view the operation of the image recognition model deployed in the agricultural machinery 101, and can also interact with the server 104 through the display screen 102 in the agricultural machinery 101 to receive or send messages, etc.

[0032] The server 104 may be a server that provides various services, such as training and optimizing an image recognition model deployed in the agricultural machinery 101 (only as an example).

[0033] It should be noted that the agricultural machinery field image processing method based on image recognition technology provided in the embodiment of the present disclosure can generally be executed by the agricultural machinery equipment 101. Accordingly, the agricultural machinery field image processing device based on image recognition technology provided in the embodiment of the present disclosure can generally be set in the server 104.

[0034] It should be understood that Figure 1 The number of agricultural machinery equipment, networks and servers in the embodiment is only for illustration. Any number of terminal equipment, networks and servers may be provided according to the implementation requirements.

[0035] Figure 2 The flowchart of the method for processing images of agricultural machinery operation fields based on image recognition technology according to an embodiment of the present disclosure is schematically shown.

[0036] like Figure 2 As shown, the embodiment 200 includes operations S210 to S260.

[0037] In operation S210, a plurality of first crop test images of a first test area in a field area are acquired by an image acquisition unit of the agricultural machinery.

[0038] In operation S220, image recognition is performed on the plurality of first crop test images based on the image recognition model deployed on the agricultural machinery equipment to obtain first image recognition results of each of the plurality of first crop test images.

[0039] In operation S230, a supplementary data set for the field area is acquired through an image acquisition unit according to the model supplementary parameters obtained after analyzing the plurality of first image recognition results.

[0040] In operation S240 , the supplementary data set is sent to the cloud so that the cloud optimizes the image recognition model according to the supplementary data set.

[0041] In operation S250, the original parameters of the image recognition model deployed on the agricultural machinery are replaced with the optimized model parameters received from the cloud to obtain an optimized image recognition model.

[0042] In operation S260, image recognition is performed on a plurality of crop images collected in real time based on the optimized image recognition model to obtain a plurality of second image recognition results, so as to control agricultural machinery to perform field operations according to the second image recognition results.

[0043] According to an embodiment of the present disclosure, agricultural machinery refers to a movable vehicle that can carry a field operation device, such as a tractor, etc. Based on the type of field operation device carried by the agricultural machinery, the agricultural machinery can perform a variety of field operations. For example, agricultural machinery equipped with a plant protection waterway can perform field crop irrigation operations, field weeding operations, etc., and agricultural machinery equipped with a seed drill can perform field seeding operations, field seeding operations, etc.

[0044] Among them, the field operations performed by agricultural machinery include field operations based on image recognition technology, that is, the image recognition results based on image recognition technology trigger the agricultural machinery to automatically start, end, and adjust the field operations.

[0045] According to an embodiment of the present disclosure, an image acquisition unit is provided on the agricultural machinery, and as the agricultural machinery moves, the image acquisition unit can acquire crop images at different locations and in different areas of the field in real time. The image acquisition unit can automatically acquire crop images.

[0046] Since the actual scene changes in the field area will vary with the soil, climate, light, crop cultivation method, crop growth period, etc., for example, the growth of crops two days apart is different, and the growth of crops during the day and at night is different. Therefore, the method of performing image analysis on crop images pre-collected by equipment such as drones does not meet the real-time requirements in the agricultural field. The embodiment of the present disclosure deploys an image recognition model based on image recognition technology on agricultural machinery and equipment, so that the agricultural machinery and equipment can perform image analysis, such as image recognition, based on the crop images collected in real time, and then trigger the execution of field operations in real time through the image recognition results, so as to achieve synchronous image collection and synchronous execution of field operations by agricultural machinery and equipment.

[0047] According to the embodiments of the present disclosure, the image recognition model deployed on the agricultural machinery can be based on pre-trained ones. However, since different periods, different crops, different lands, etc. will affect the model results, when performing field operations in the field area, it is necessary to collect multiple first crop test images in real time to determine whether the image recognition model deployed on the agricultural machinery needs to be optimized. If optimization is required, it is optimized according to the actual scene of the current field area, so as to further improve the accuracy of field operations for the field area.

[0048] The field area may be a field area divided into preset land units, such as one acre of land. The first test area in the field area may be any area in the field area, such as a row of crops.

[0049] When the agricultural machinery equipment is traveling in the first test area, multiple first crop test images can be collected by the image acquisition unit, and the field operation device can be controlled to perform field operations based on the multiple first crop test images. It should be noted that the multiple first crop test images can control the field operation device to perform field operations in the first test area, and can also be used to optimize the image recognition model based on the actual scene of the first test area, improve the quality of field operations in other areas of the field area, and there is no need to perform field operations in the first test area again.

[0050] According to an embodiment of the present disclosure, the image recognition model may be constructed based on an existing image recognition algorithm, for example, an object detection algorithm.

[0051] The multiple first image recognition results can reflect the recognition accuracy and adaptability of the image recognition model to the field area. Therefore, after analyzing the multiple first image recognition results, it can be determined whether the image recognition model needs to be optimized. After analyzing the multiple first image recognition results, if the recognition accuracy of the image recognition model meets a predetermined condition, such as a certain threshold, the image recognition model may not be optimized, and the agricultural machinery equipment may be controlled to perform field operations in the field area directly based on the image recognition results of the field area by the image recognition model. If the recognition accuracy of the image recognition model does not meet the predetermined conditions, the model supplementary parameters are determined so that a supplementary data set for the field area is obtained through the image acquisition unit.

[0052] According to an embodiment of the present disclosure, the supplementary data set is the crop images of the field area, which are used to optimize the image recognition model specifically for the actual scene of the field area.

[0053] The agricultural machinery can send the supplementary data set to the cloud through a network communication connection, and the cloud stores the original parameters of the image recognition model deployed on the agricultural machinery. The embodiment of the present disclosure uses a cloud with greater computing power to achieve rapid optimization of the image recognition model, and then returns the optimized model parameters obtained after the cloud optimizes the image recognition model to the agricultural machinery, so as to avoid the lack of computing power of the agricultural machinery affecting the real-time operation in the field, and avoid the cost increase caused by deploying more computing power on the agricultural machinery.

[0054] Agricultural machinery can use the optimized model parameters received from the cloud to replace the original parameters to obtain an optimized image recognition model.

[0055] After obtaining the optimized image recognition model, when the agricultural machinery is traveling in the field area, multiple crop images can be collected by the image acquisition unit, and the field operation device can be controlled to perform field operations normally based on the multiple crop images.

[0056] The embodiments of the present disclosure deploy image recognition models on agricultural machinery used to perform field operations, and can obtain image recognition models suitable for different field areas through real-time acquisition and real-time optimization, and simultaneously use the optimized image recognition models to perform field operations in real time. As a result, the embodiments of the present disclosure can realize the acquisition and analysis of crop images while performing field operations, thereby improving the in-depth utilization of image recognition technology in field operations, improving the efficiency of automated execution of field operations, improving the quality of field operations, and meeting the timeliness and accuracy requirements of field operations.

[0057] According to an embodiment of the present disclosure, the method also includes: displaying multiple first image recognition results to an operator through a display unit of the agricultural machinery equipment; determining label information of each of the multiple first image recognition results based on the operator's interactive operation on the display unit; and determining model supplement parameters based on the multiple label information.

[0058] According to an embodiment of the present disclosure, the display unit may be a touch-interactive display screen on the agricultural machinery. The operator can view the image recognition results of the image recognition model through the display unit, and can also receive prompt information of the warning area with problems. In addition, the operator can also operate and view on the display screen, judge the warning area and make timely optimization adjustments to the operation, or record the problem.

[0059] For example, after displaying multiple first image recognition results to the operator through the display unit of the agricultural machinery, the operator can view the first image recognition results, such as the framed crops and weeds. The operator can also view the first image recognition results to see whether all weeds and crops in the first crop test image are identified, and then determine the label information of each first image recognition result through interaction with the display screen. The label information includes a label indicating whether the recognition accuracy meets the requirements.

[0060] According to an embodiment of the present disclosure, the label information includes an identified target or a misidentified target, and the model supplementary parameters are determined according to the plurality of label information, including at least one of the following:

[0061] Determine environmental characteristics for the field area according to the identification targets in the plurality of tag information; determine a first number of model supplementary parameters when the environmental characteristics meet a predetermined condition; and determine a second number of model supplementary parameters when the environmental characteristics do not meet the predetermined condition;

[0062] When the number of misidentified targets in the plurality of label information meets a predetermined threshold, a first number of model supplementary parameters is determined.

[0063] According to an embodiment of the present disclosure, the environmental feature includes at least one of the following: the type of crops in the field area, the number of identified targets, and the recognition accuracy rate of each identified target.

[0064] According to the embodiments of the present disclosure, since the actual scenes in the field area are relatively complex and vary greatly over time, the label information determined based on the interactive operation of the operator can reflect the actual environmental characteristics of the current field area.

[0065] In a specific embodiment, when the tag information is an identification target, the environmental characteristics of the entire field area can be determined based on multiple tag information. The identification target can be crops in the current field area or weeds. For example, based on the identification target in each tag information, the environmental characteristics of the field area, such as crop production conditions and weed growth conditions, are determined.

[0066] If the environmental characteristics meet the predetermined conditions, a first number of model supplementary parameters is determined. If the environmental characteristics do not meet the predetermined conditions, a second number of model supplementary parameters is determined. The first number is less than the second number.

[0067] For example, when the environmental feature is the type of crops in the field area, the environmental feature satisfies the predetermined condition, that is, the crop type is the predetermined crop type. Generally, for predetermined crop types, such as rice, etc., which have a special growth environment, the image recognition model is relatively easy to identify, so a smaller number of first-number model supplementary parameters can be used for model optimization to reduce the amount of data processing; for crops such as sweet potatoes and potatoes, their land environment is relatively complex and difficult to distinguish from weeds, and the accuracy of the image recognition model is easily affected, so a larger number of second-number model supplementary parameters can be used for model optimization.

[0068] In the case where the environmental feature is the number of identified targets, the environmental feature satisfies the predetermined condition, that is, the number of identified targets is less than the predetermined number threshold. If there are fewer targets to be identified in the field area, a smaller first number of model supplementary parameters is used for model optimization; otherwise, a larger second number of model supplementary parameters is used for model optimization.

[0069] In the case where the environmental feature is the recognition accuracy of each recognition target, the environmental feature satisfies the predetermined condition, that is, the overall recognition accuracy reaches the recognition threshold, or the recognition accuracy of the crop reaches the recognition threshold. If the overall recognition accuracy reaches the recognition threshold, a smaller first number of model supplementary parameters are used for model optimization; otherwise, a larger second number of model supplementary parameters are used for model optimization.

[0070] Since image recognition may be affected by external factors in some cases, causing the recognition results to fluctuate dramatically, the overall recognition accuracy can be effectively evaluated by calculating the average of multiple image recognitions.

[0071] In another specific embodiment, when the tag information is a misidentified target, the model supplementary parameters can be determined according to whether the number of misidentified targets meets a predetermined threshold. When the number of misidentified targets meets the predetermined threshold, it indicates that the current environment features are relatively simple, and a smaller first number of model supplementary parameters can be used.

[0072] According to an embodiment of the present disclosure, the model supplementary parameters include at least one of the following: shooting frequency, zoom parameter, zoom mode, image resolution, and image ratio.

[0073] In the above embodiment, the first number or the second number of model supplementary parameters may be randomly determined from the model supplementary parameters.

[0074] In another specific embodiment, a selection page may also be displayed to the operator, and the operator may independently select the first number or the second number of model supplementary parameters through the selection page.

[0075] For example, the first number may be 3 and the second number may be 5, that is, the second number is all the model supplementary parameters.

[0076] According to the embodiments of the present disclosure, the shooting frequency is the shooting frequency of the image acquisition unit. When determining the model supplementary parameters, the shooting frequency can be adjusted according to the preset shooting frequency, or the shooting frequency can be customized according to the interaction of the operator. It should be noted that the embodiments of the present disclosure can perform image recognition and field operations in real time through agricultural machinery and equipment, and the field operations are triggered by image recognition. If the model operation capability does not match the image acquisition unit, the shooting frequency will be low under high-speed operation, and the image recognition will not keep up with the vehicle speed, resulting in inaccurate model results. Therefore, the customized or preset shooting frequency matches the model operation capability of the image recognition model.

[0077] According to an embodiment of the present disclosure, the zoom parameter may be a custom or predetermined focal length, and the zoom mode may include manual zoom and automatic zoom.

[0078] The image resolution is generally 1920*1280 and 1280*720, and the image ratio is 16:9 and 4:3.

[0079] According to the embodiments of the present disclosure, shooting frequency, zoom parameters, zoom mode, image resolution, and image ratio are all model supplement parameters in the image acquisition stage, which can collect images matching the actual scenes of the current field area in real time, constitute a supplementary data set, and optimize the image recognition model.

[0080] Specifically, based on the model supplementary parameters obtained after analyzing the multiple first image recognition results, a supplementary data set for the field area is obtained through the image acquisition unit, including: adjusting the image acquisition parameters of the image acquisition unit according to the model supplementary parameters; and using multiple second crop test images of the second test area re-acquired based on the image acquisition parameters as the supplementary data set.

[0081] For example, through the display screen of agricultural machinery, the operator can see multiple first image recognition results output by the image recognition model. At the same time, the operator can also view the training set information of the image recognition model, such as the parameters of the training set, the type of crops targeted, the number of targets to be recognized, and other parameter information. When the recognition rate accuracy of the first image recognition result is not high, such as less than 80%, it means that the generalization of the model is insufficient and more images are needed. Thus, the operator can set parameters such as shooting frequency, automatic zoom, manual zoom, image resolution, image ratio, etc., and complete the image acquisition of multiple second crop test images in the second test area while performing field operations through agricultural machinery. Afterwards, the operator can also select the storage directory through the display screen to compile the collected multiple second crop test images into the supplementary data set for model optimization.

[0082] Therefore, when optimizing the model, targeted image acquisition can be performed according to the supplementary parameters of each model, which facilitates the use of different model supplementary parameters for image enhancement in the cloud, so as to ensure that the optimized image recognition model can be more suitable for the actual scenes in the current field area.

[0083] The embodiments of the present disclosure increase the supplementary data set by adding model supplementary parameters, which can make the model more generalizable and improve its adaptability to the current field area, thereby improving both the model accuracy and recall rate when re-optimizing the model.

[0084] In another specific embodiment, the model supplementation parameters may also include model identification parameters for the model identification stage.

[0085] Specifically, after performing image acquisition and image recognition through operation S210 and operation S220, the operator can view the operation status of the model and multiple first image recognition results. According to the multiple first image recognition results, the accuracy distribution of the model image recognition and whether other targets are detected incorrectly can be seen. At this time, the operator can adjust the accuracy of the model through interactive operations. For example, if there are many targets that need to be identified in the field, the model recognition parameters are increased, otherwise the model recognition parameters are reduced. For example, when the model is running, if the accuracy of the recognition box marking the crops is low, the model recognition parameters can be appropriately reduced. This method is suitable for identifying image recognition models with larger field targets.

[0086] In the case of adjusting the model recognition parameters, the image recognition model after the model recognition parameters are adjusted can be directly used as the optimized image recognition model.

[0087] According to the embodiments of the present disclosure, the model recognition parameter may be understood as a threshold parameter used in the output layer of the image recognition model to determine the type of the recognition target.

[0088] According to an embodiment of the present disclosure, the optimization of the image recognition model according to the supplementary data set in the cloud in operation S240 includes the following four steps.

[0089] Step 1: The cloud randomly classifies the training set and the test set according to the received supplementary data set, and classifies all images in the supplementary data set into two sets. Specifically, the supplementary data set includes a supplementary data subset corresponding to each model supplementary parameter. During classification, all images in each supplementary data subset need to be classified into a training set and a test set.

[0090] Step 2: The cloud will perform image enhancement processing based on the number and priority of the model supplementary parameters to generate a new test image. For example, image enhancement processing can be operations such as cropping and denoising.

[0091] Step 3: The cloud automatically optimizes the model based on the training cycle parameters configured by the user and outputs the optimized model parameters to the agricultural machinery.

[0092] Step 4: If the operator is satisfied with the optimization result, he can use the optimized model parameters received from the cloud to replace the original parameters of the image recognition model deployed on the agricultural machinery to obtain the optimized image recognition model and use it.

[0093] For step 3, the parameters of the number of training rounds can be set, such as 10-30 training rounds. Then, the trained model is further verified in the field until the recognition accuracy meets the requirements, and then the final optimized model parameters are output to the agricultural machinery.

[0094] According to an embodiment of the present disclosure, original parameters of an image recognition model deployed on agricultural machinery are replaced with optimized model parameters received from the cloud to obtain an optimized image recognition model, including: replacing original parameters with intermediate model parameters received from the cloud to obtain an intermediate image recognition model, wherein the intermediate model parameters are obtained after the cloud performs a preset number of optimization operations on the image recognition model using a supplementary data set; based on the intermediate image recognition model, performing image recognition on multiple third crop test images of a third test area to obtain third image recognition results of each of the multiple third crop test images; when the multiple third image recognition results meet the accuracy conditions of the agricultural machinery, using the intermediate model parameters as optimized model parameters and using the intermediate image recognition model as the optimized image recognition model.

[0095] According to an embodiment of the present disclosure, after the intermediate image recognition model is obtained, it can be determined whether the intermediate image recognition model is an image recognition model adapted to the current field area based on the third image recognition result of the third test area.

[0096] According to an embodiment of the present disclosure, a verification time parameter can be set, with a minimum of 5 minutes and a maximum of 1 hour. During the optimization period, the verification time parameter of the training set is set, and the recognition accuracy within the verification time parameter is calculated as a reference for whether the model meets the optimization standard.

[0097] According to the embodiments of the present disclosure, since the computing power consumption of the model on agricultural machinery equipment cannot be accurately known before deployment, the type and quantity of image acquisition units deployed on each agricultural machinery equipment are different, and the speed of each model performing different field operations is also different. Therefore, it is necessary to test the computing power matching between the model and the agricultural machinery equipment in order to derive the best usage of the model.

[0098] Specifically, the recognition speed of the image recognition model during image recognition is obtained; when the recognition speed is less than a preset speed, the operating parameters of the agricultural machinery are adjusted, wherein the preset speed is related to the vehicle speed of the agricultural machinery.

[0099] In this embodiment, the preset speed can be based on the current vehicle speed when driving 1 meter, and the recognition speed is not less than 10 pictures and not more than 20 pictures. If it is less than 10 pictures, it means that the computing power is insufficient and the computing power condition is not met. If it is more than 20 pictures, it means that the computing power is surplus and other adaptive changes can be made.

[0100] After the model is deployed on agricultural machinery, image acquisition, image recognition, and field operation tests can be performed simultaneously in the first test area / the second test area / the third test area. The real-time status of the model operation, the status of the marked identified crops, and the current model recognition speed can be viewed, such as XX images / mm seconds.

[0101] Therefore, when the recognition speed is lower than the preset speed, the operating parameters of the agricultural machinery are adjusted so that the model matches the computing power of the agricultural machinery.

[0102] According to an embodiment of the present disclosure, the operation parameters include at least one of the following: the speed of the agricultural machinery, the shooting frequency of the image acquisition unit, the number of image acquisition units in working state,

[0103] For example, when the recognition speed is less than the preset speed, the operator can reduce the vehicle speed, or optimize the model recognition parameters, such as reducing the number of recognized targets, or reducing the number of the image acquisition units in operation, so as to reduce the load of each image acquisition hardware. If the speed is higher than 20 images, the number of the image acquisition units in operation can be increased, or the vehicle speed can be increased, or the number of recognized targets can be increased.

[0104] Figure 3 The application scenario of the method for processing images of agricultural machinery operations in the field based on image recognition technology according to an embodiment of the present disclosure is schematically illustrated.

[0105] like Figure 3 As shown, the crops in the right column are used as the first test area 301. After the agricultural machinery 310 passes through the first test area 301, multiple first crop test images can be obtained. According to the first image recognition results of the multiple first crop test images, the model supplementary parameters can be determined. When performing field operations in the second test area 302, a supplementary data set can be obtained according to the model supplementary parameters. The supplementary data set is sent to the cloud 320 for model optimization, and after the optimization is completed, the optimized model parameters are received from the cloud, and the optimized model parameters are used to replace the original parameters to obtain the optimized image recognition model deployed on the agricultural machinery. Afterwards, the agricultural machinery 310 can continue to perform field operations on other areas of the field area based on the optimized image recognition model.

[0106] For ease of understanding, the present solution will be described below using an example.

[0107] The image acquisition unit is fixed on the agricultural machinery and connected to the host computer of the agricultural machinery so as to interact through the display screen of the host computer. For example, the image recognition model can be configured and optimized through the display screen. The configuration of the image recognition model includes parameter configuration, accuracy detection, cloud training, model selection, model deletion, etc.

[0108] After completing the above configuration, select New Job, Historical Job, and Last Job to complete the configuration of job startup. To create a new job, you need to configure the name of the current job, select the image recognition model, and after completing the model parameter configuration, select the image processing method, such as storage or non-storage. After selecting storage, select record recognition points + original image to store only the original image. After completing the configuration, you can perform field operations. If you select Historical Job, you need to select the name of the historical job and then enter the field operation. If you select Last Job, the configuration parameters of the most recent job are used by default.

[0109] After the operation starts, the display screen will show the crop images captured by the image acquisition unit. In addition, the image recognition model will also perform image recognition and display on the crop images, and mark the identified abnormal information. After seeing the abnormal information, the operator can handle it based on experience.

[0110] After completing the detection of the first test area based on the above operations, if the model recognition accuracy is poor, the model recognition parameters can be adjusted. If there are many targets to be recognized in the field, the model recognition parameters can be increased, otherwise the model recognition parameters can be reduced.

[0111] If adjusting the model recognition parameters still cannot ensure that the recognition accuracy reaches the application effect, it means that the generalization of the model needs to be improved. Therefore, the device image collection function can be used to collect supplementary data sets based on the model supplementary parameters and upload them to the cloud. After starting the device image collection function, the operator can also customize the model supplementary parameters or automatically select the first number / second number of model supplementary parameters based on the default options.

[0112] Afterwards, the cloud automatically optimizes the model according to the training cycle parameters configured by the user, and deploys the optimized model on the agricultural machinery. In this process, the optimized model parameters output by the cloud can be directly used to replace the original parameters to obtain an optimized image recognition model; the optimized model parameters can also be used to replace the original parameters to obtain an intermediate image recognition model, and actual testing of the third test area can be used to determine whether to use the intermediate image recognition model as the optimized image recognition model.

[0113] Finally, after the optimization is completed, image recognition can be performed on multiple crop images collected in real time based on the optimized image recognition model to obtain multiple second image recognition results, so as to control agricultural machinery to perform field operations according to the second image recognition results.

[0114] According to an embodiment of the present disclosure, an agricultural machinery field image processing device based on image recognition technology includes: a first acquisition module, which is used to acquire multiple first crop test images of a first test area in a field area through an image acquisition unit of agricultural machinery equipment; a first recognition module, which is used to perform image recognition on multiple first crop test images based on an image recognition model deployed on the agricultural machinery equipment, and obtain first image recognition results of each of the multiple first crop test images; a second acquisition module, which is used to acquire a supplementary data set for the field area through the image acquisition unit according to model supplementary parameters obtained after analyzing the multiple first image recognition results; a first optimization module, which is used to send the supplementary data set to the cloud, so that the cloud optimizes the image recognition model according to the supplementary data set; a second optimization module, which is used to replace the original parameters of the image recognition model deployed on the agricultural machinery equipment with the optimized model parameters received from the cloud, and obtain an optimized image recognition model; a second recognition module, which is used to perform image recognition on multiple crop images collected in real time based on the optimized image recognition model, and obtain multiple second image recognition results, so as to control the agricultural machinery equipment to perform field operations according to the second image recognition results.

[0115] It should be noted that the device part in the embodiments of the present disclosure corresponds to the method part in the embodiments of the present disclosure. The description of the device part specifically refers to the method part and will not be repeated here.

[0116] According to an embodiment of the present disclosure, any multiple modules among the first acquisition module, the first identification module, the second acquisition module, the first optimization module, the second optimization module, and the second identification module can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0117] According to an embodiment of the present disclosure, at least one of the first acquisition module, the first identification module, the second acquisition module, the first optimization module, the second optimization module, and the second identification module may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by an appropriate combination of any of them. Alternatively, at least one of the first acquisition module, the first identification module, the second acquisition module, the first optimization module, the second optimization module, and the second identification module may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function may be executed.

[0118] Figure 4 A block diagram of an electronic device suitable for a method for processing images of agricultural machinery operations in a field based on image recognition technology according to an embodiment of the present disclosure is schematically shown.

[0119] like Figure 4 As shown, the electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 to a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include an onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0120] In RAM 403, various programs and data required for the operation of electronic device 400 are stored. Processor 401, ROM 402 and RAM 403 are connected to each other via bus 404. Processor 401 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 402 and / or RAM 403. It should be noted that the program can also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0121] According to an embodiment of the present disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to the bus 404. The electronic device 400 may further include one or more of the following components connected to the input / output I / O interface 405: an input portion 406 including a keyboard, a mouse, etc.; an output portion 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 408 including a hard disk, etc.; and a communication portion 409 including a network interface card such as a LAN card, a modem, etc. The communication portion 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage portion 408 as needed.

[0122] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0123] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0124] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the above method provided by the embodiment of the present disclosure.

[0125] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 401. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0126] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 409, and / or installed from the removable medium 411. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0127] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0128] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0129] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0131] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above description is only a specific embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for processing images of agricultural machinery operation fields based on image recognition technology, characterized in that: The method comprises: Acquiring a plurality of first crop test images of a first test area in a field area through an image acquisition unit of the agricultural machinery; Based on the image recognition model deployed on the agricultural machinery, performing image recognition on the plurality of first crop test images to obtain first image recognition results of the plurality of first crop test images; According to the model supplementary parameters obtained after analyzing the plurality of first image recognition results, obtaining a supplementary data set for the field area through the image acquisition unit; Sending the supplementary data set to the cloud so that the cloud optimizes the image recognition model according to the supplementary data set; Replacing original parameters of the image recognition model deployed on the agricultural machinery with the optimized model parameters received from the cloud to obtain an optimized image recognition model; and Based on the optimized image recognition model, image recognition is performed on multiple crop images collected in real time to obtain multiple second image recognition results, so as to control the agricultural machinery to perform field operations according to the second image recognition results.

2. The method according to claim 1, characterized in that: Also includes: displaying the plurality of first image recognition results to an operator through a display unit of the agricultural machinery; Determining label information of each of the plurality of first image recognition results according to the interactive operation of the operator on the display unit; The model supplementary parameters are determined according to the plurality of label information.

3. The method according to claim 2, characterized in that The tag information includes an identified target or a misidentified target, and the determining of the model supplementary parameter according to the plurality of tag information includes at least one of the following: Determine the environmental characteristics for the field area according to the identification targets in the plurality of tag information; determine a first number of the model supplementary parameters when the environmental characteristics meet a predetermined condition; and determine a second number of the model supplementary parameters when the environmental characteristics do not meet the predetermined condition; When the number of misidentified targets in the plurality of tag information meets a predetermined threshold, a first number of the model supplementary parameters is determined.

4. The method according to claim 3, characterized in that The environmental characteristics include at least one of the following: the type of crops in the field area, the number of identified targets, and the recognition accuracy of each identified target.

5. The method according to any one of claims 1 to 4, characterized in that: The model supplementary parameters include at least one of the following: shooting frequency, zoom parameter, zoom mode, image resolution, and image ratio.

6. The method according to claim 1, characterized in that The method of acquiring a supplementary data set for the field area through the image acquisition unit according to the model supplementary parameters obtained after analyzing the plurality of first image recognition results includes: adjusting the image acquisition parameters of the image acquisition unit according to the model supplementary parameters; A plurality of second crop test images of a second test area reacquired based on the image acquisition parameters are used as the supplementary data set.

7. The method according to claim 1, characterized in that The method of replacing original parameters of the image recognition model deployed on the agricultural machinery equipment with the optimized model parameters received from the cloud to obtain an optimized image recognition model includes: Replacing the original parameters with intermediate model parameters received from the cloud to obtain an intermediate image recognition model, wherein the intermediate model parameters are obtained after the cloud performs a preset number of optimization operations on the image recognition model using the supplementary data set; Based on the intermediate image recognition model, performing image recognition on a plurality of third crop test images in a third test area to obtain third image recognition results of each of the plurality of third crop test images; In the case that a plurality of the third image recognition results satisfy the accuracy condition of the agricultural machinery equipment, the intermediate model parameters are used as the optimization model parameters, and the intermediate image recognition model is used as the optimized image recognition model.

8. The method according to claim 1, characterized in that The method further comprises: Obtaining a recognition speed when the image recognition model performs image recognition; When the identified speed is less than the preset speed, the operating parameters of the agricultural machinery are adjusted, wherein the preset speed is related to the vehicle speed of the agricultural machinery.

9. The method according to claim 8, characterized in that The operation parameter includes at least one of the following: the speed of the agricultural machinery, the shooting frequency of the image acquisition unit, and the number of the image acquisition units in working state.

10. An agricultural machine comprising: The main body of agricultural machinery is used to perform field operations; An image acquisition unit, arranged on the agricultural machinery main body, for acquiring images; Electronic equipment, including: One or more processors; a memory for storing one or more computer programs; characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

Images