Farm work type detection method, equipment, medium and product

Abstract

The embodiment of the invention relates to the technical field of agricultural management, and discloses a farming type detection method and device, a medium and a product. Comprising the following steps: acquiring a farmland video, and processing the farmland video through a YOLOv3 model to obtain a dynamic 2D region of each operator; identifying the figure image of the dynamic 2D area through a FaceNet network model to obtain identity information of the operator; recognizing the figure image of the dynamic 2D area through a farming action recognition model to obtain action information of the operator; the farming action recognition model adopts an X-CLIP architecture and comprises a video encoder and a text encoder; and obtaining a farming action record according to the identity information and the action information of the operator. The technical problem that agricultural activity behaviors cannot be effectively monitored can be at least solved.

Description

Technical Field

[0001] The present application relates to the field of agricultural management technology, and in particular to a method, equipment, medium and product for detecting farming types. Background Art

[0002] With the development of modern agriculture, the demand for precision and intelligence in agricultural management is growing. In agricultural activities, monitoring and recording the behavior of operators plays an important role in optimizing agricultural management. However, the traditional method of monitoring agricultural actions mainly relies on manual recording, which is not only time-consuming and labor-intensive, but also easily affected by human factors, resulting in the inability to guarantee the accuracy and consistency of the data, especially in complex farmland operation environments, where manual recording is difficult to provide accurate operation behavior data. Existing agricultural management technologies are mostly focused on the monitoring and analysis of crops, such as soil moisture, temperature, and crop growth monitoring, but research on operator behavior monitoring is relatively limited. Due to the diversity of agricultural activities, complex scenarios, and high mobility of personnel during operations, how to effectively monitor personnel behavior and accurately identify their agricultural operations has become a major challenge in agricultural automation management.

[0003] In recent years, with the rapid development of computer vision and deep learning technologies, video surveillance technology has been widely used in various behavior recognition scenarios, such as security monitoring, sports action analysis and other fields. In particular, the action recognition model based on deep learning can automatically identify the action categories of people in the video by analyzing video data, providing a new solution for behavior monitoring in different scenarios. However, although action recognition technology has been applied in other fields, it still faces certain difficulties in applying it to agricultural scenarios. Agricultural operations are highly professional, with many types of actions and obvious differences in details. For example, there are large differences in actions between operations such as transplanting, sowing, fertilizing, and spraying. At the same time, the complex farmland environment, light, weather, and the diversity of operators also bring challenges to the accurate identification of operation actions. Summary of the invention

[0004] One purpose of the present application is to provide a method, device, medium and product for detecting agricultural activity types, at least to solve the technical problem of being unable to effectively monitor agricultural activity behaviors.

[0005] To achieve the above objectives, some embodiments of the present application provide the following aspects:

[0006] In the first aspect, some embodiments of the present application also provide a method for detecting an agricultural type, including obtaining a farmland video, processing the farmland video through a YOLOv3 model, and obtaining a dynamic 2D area of ​​each worker; recognizing a person image in the dynamic 2D area through a FaceNet network model to obtain identity information of the worker; recognizing a person image in the dynamic 2D area through an agricultural action recognition model to obtain action information of the worker; the agricultural action recognition model adopts an X-CL IP architecture and includes a video encoder and a text encoder, the video encoder is used to extract a video feature vector, and the text encoder is used to extract a text feature vector; and obtaining an agricultural action record based on the identity information and action information of the worker.

[0007] In a second aspect, some embodiments of the present application further provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.

[0008] In a third aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method as described above.

[0009] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program / instruction, which implements the steps of the method described above when executed by a processor.

[0010] Compared with the related technology, the solution provided in the embodiment of the present application realizes accurate recognition of the identity and actions of farm workers by combining the YOLOv3 model and the FaceNet network; compared with the traditional agricultural recording method that relies on manual labor, the action recognition model based on deep learning can automatically generate agricultural action records; real-time and accurate identification of agricultural operations, automatic recording of the actions and time information of workers, helps to optimize the agricultural management process, reduce dependence on manual monitoring, and significantly improve the efficiency of agricultural management. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0012] Figure 1 A schematic diagram of a method for detecting farming types according to an embodiment of the present application;

[0013] Figure 2 A schematic diagram of a flow chart of another farming type detection method provided according to an embodiment of the present application;

[0014] Figure 3 It is a structural schematic diagram of a farming type detection system provided according to an embodiment of the present application;

[0015] Figure 4 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] First embodiment

[0018] The first embodiment of the present application relates to a method for detecting farming types. Figure 1 As shown, the method may include the following steps:

[0019] S101, obtaining a farmland video, and processing the farmland video using a YOLOv3 model to obtain a dynamic 2D region of each operator;

[0020] S102, identifying the person image in the dynamic 2D area by using a FaceNet network model to obtain the identity information of the operator;

[0021] S103, recognizing the person image in the dynamic 2D area by using a farming action recognition model to obtain action information of the operator; the farming action recognition model adopts an X-CL IP architecture and includes a video encoder and a text encoder, the video encoder is used to extract a video feature vector, and the text encoder is used to extract a text feature vector;

[0022] S104, obtaining agricultural action records according to the identity information and action information of the operator.

[0023] Farming types refer to the actions people are taking in agricultural production activities, including transplanting, sowing, fertilizing, spraying, harvesting, and other actions. During agricultural management, the farmland is continuously and effectively observed and recorded through detection devices (such as cameras) around the farmland. Different types of operations in farming operations are automatically identified based on video surveillance and deep learning technology. Farmland operation videos are collected and processed in real time, and the identities of operators and the farming actions they perform are automatically identified, and then farming action records are generated.

[0024] For step S101, the agricultural operation is continuously monitored by video equipment deployed in the farmland to obtain the farmland operation video of the operator. Then, the pre-trained YOLOv3 model is used to detect the characters in the video. The YOLOv3 model is an object detection model based on deep learning, which can detect the position of the operator in the video in real time. The model represents each character in the video frame with a dynamic 2D bounding box and outputs the dynamic 2D area of ​​each operator, thereby locating the specific position of each operator in the video.

[0025] For step S102, after obtaining the dynamic 2D area of ​​each operator, the FaceNet network model is used to identify the person in the 2D area. The FaceNet network is a deep learning model based on face recognition, which can distinguish different operators by extracting facial feature vectors. The model uses the Zeiler & Fergus structure, with a fixed-size image as input and a feature vector of the person as output. By calculating the Euclidean distance between two feature vectors, the FaceNet network can accurately identify the identity of the operator.

[0026] For step S103, after the identity of the operator is determined, the agricultural action recognition model based on the X-CLIP architecture is used to detect the operator's actions. X-CLIP is a video action recognition model that combines a video encoder and a text encoder to achieve recognition of character actions in the video. The video encoder is used to extract video feature vectors from the dynamic 2D area of ​​the operator, while the text encoder is used to extract corresponding text feature vectors from predefined agricultural action descriptions. By aligning the video feature vector with the text feature vector, the model can accurately determine the type of agricultural operation currently being performed by the operator, such as transplanting, sowing, fertilizing, spraying, etc.

[0027] For step S104, the corresponding agricultural action record is automatically generated according to the identified operator identity information and the detected operator action information. The record includes key information such as the operator's identity, action type, and operation time, providing a basis for subsequent agricultural management and decision-making. Through this method, the operation status in the farmland can be monitored in real time to ensure the reasonable planning and execution of agricultural operations.

[0028] It is not difficult to find that compared with the relevant technology, the solution provided by the embodiment of the present application realizes real-time identity recognition and action detection of farm workers through the combination of YOLOv3 and FaceNet network. The introduction of X-CLIP architecture further improves the accuracy and stability of farm action recognition, and can automatically generate farm operation records in complex farm operation scenes, providing technical support for the intelligent and precise management of agricultural production.

[0029] Second embodiment

[0030] The second embodiment of the present application relates to a method for detecting farming types. The second implementation is an improvement on the first embodiment, and the specific improvements are:

[0031] like Figure 2 As shown, real-time farmland videos are obtained by deploying cameras in the farmland environment. The farmland videos shown contain dynamic activity scenes of multiple workers, and subsequent farming action recognition and recording are performed based on these farmland videos.

[0032] The acquired farmland videos are processed and the YOLOv3 model is used to detect the workers in the videos. The YOLOv3 model is an efficient deep learning target detection algorithm that can identify objects in video frames and locate their positions in real time. Through this model, a dynamic 2D area can be generated for each worker. The area will be updated in the video as the worker moves, and the location of the worker can be marked.

[0033] After locating the dynamic 2D area of ​​each operator, the FaceNet network model is used to recognize the face in the 2D area. The FaceNet network model uses the Zeiler & Fergus structure, which has 6 convolutional layers, with a fixed length and width image x as input and a feature vector of the image as output. The distance between two eigenvectors is their Euclidean distance. To normalize the distance metric, the lengths of all eigenvectors are scaled to 1, that is, to ensure that ||f(x)||2=1.

[0034] Euclidean distance, for two vectors y = [y1, y2, ..., y n ],z=[z1,z,…,z n ], the square of the Euclidean distance between the two is:

[0035]

[0036] For the picture corresponding to the farm worker p (target), which corresponds to all other images of this person in the training data The distance (positive sample) is greater than the images corresponding to all other persons in the training data (negative sample) distance. The formal expression is:

[0037]

[0038] α is the maximum marginal distance between positive samples and negative samples, which is a pre-set parameter.

[0039] T is all possible (target, positive sample, negative sample) triplets in the training set

[0040] Therefore, the loss function of the entire FaceNet network is:

[0041]

[0042] N is the total number of people, and the other symbols are the same as above. This loss function will be minimized during training.

[0043] In order to ensure the efficiency of calculation, S image samples are randomly selected for each person for actual calculation. In the actual calculation, Z frames are randomly selected from the previously obtained dynamic 2D area of ​​each person as the input images for face recognition, and the image size is adjusted to the acceptable size of the FaceNet network. For the kth person, its score is considered to be:

[0044]

[0045] That is, the sum of the squares of the Euclidean distances between the Z video frames and the S k-th person's image samples. The k that makes this score the lowest (the smallest distance) is considered to be the final prediction answer, that is, the person recognition result is:

[0046]

[0047] Therefore, the specific identity of each character in the dynamic 2D character video can be obtained.

[0048] After completing the identification of the corresponding operators, action recognition is required; the image of each person's dynamic 2D area is input into the action recognition X-CL IP model to analyze the corresponding agricultural action. Use the X-CL IP architecture to train an agricultural action recognition model (action recognition X-CL IP), and learn to align video representations and their corresponding text representations by jointly training the video encoder and text encoder. The language image model CLIP is used as the basic model for training. The agricultural action recognition model is described in three parts below: the overall training framework, the video encoder, and the text encoder.

[0049] Overall training framework: For each video clip V and each text description C (C is an action, such as transplanting rice, watering, etc. in this example), the video V is input into the video encoder Input the text description C into the text encoder Get a video feature vector v and a text feature vector c, where:

[0050]

[0051] Then use a video-oriented description word generator To generate instance-level text representation of each video. It inputs a video feature vector v and a text feature vector c, which can be expressed as:

[0052]

[0053] Finally, a cosine similarity function is used To calculate the similarity between video and text representation, the formula is expressed as:

[0054]

[0055] in, Represents the dot product between vectors, which is the sum of the corresponding multiplication of the elements at each position of two vectors of the same size. n is the number of elements

[0056] ||a|| represents the norm of a vector, which is the sum of the squares of the elements of a vector, that is, n is the number of elements.

[0057] Video encoder: It consists of two visual transformers, the cross-frame communication transformer and the multi-frame fusion transformer. The cross-frame transformer takes the original frame as input and uses the pre-trained language image model CLIP to provide the feature representation of the frame, so that there is information transfer between frames. Then, the multi-frame fusion transformer integrates the feature representation of each frame and outputs the video feature.

[0058] Let a video clip have a resolution of H×W, T frames, and 3 color channels. Divide the tth frame into N non-overlapping small blocks Each small block is P×P pixels, and Then use a linear mapping Will Each small block in is mapped to its feature representation, where D represents the dimension after mapping. class Insert into The first part of the sequence is a vector whose parameters can be learned during training. Its state at the output of the encoder is used as part of the feature representation of the video frame.

[0059] Then the input of the cross-frame communication converter in the tth frame can be expressed as:

[0060]

[0061] Among them, e spa Represents spatial position encoding, which is a vector with different parameter values ​​for each position. There are many ways to represent it. The following is one way to represent it:

[0062]

[0063] in,

[0064]

[0065] d model Indicates the number of spatial positions it represents, in this case [x class ,Ex t,1 ,Ex T,2 ,…,Ex t,N ] represents the length of the vector

[0066] Then input it into an L c The cross-frame communication transformer (CCT) of the layer is used to obtain the frame-level feature representation h t :

[0067]

[0068]

[0069] Where l represents the layer number of CCT, Indicates the category flag x class Output:

[0070] Finally, all frame-level features are represented as H = [h1,h2,…,h T ] Input an L m The multi-frame fusion transformer (MIT) of the layer outputs the video-level feature representation v:

[0071] v = AvgPool(MIT(H+e temp ))

[0072] Among them, AvgPool represents the average pooling of the neural network; temp Indicates the position code here, which is the same as e above spa Same meaning.

[0073] Text encoder: Use the pre-trained text encoder (i.e., the text encoder of the base model CLIP) to enhance the text representation using video content. For a video language category description C, use the text encoder to obtain the feature vector c of the text description, i.e. Each language description C is a category, such as "sowing", "pesticide spraying", etc.

[0074] Then we use the video to enhance the text description. Specifically, we use a video-oriented prompt word module to enhance the text description with the video feature vector. And text feature vector c as input. Each algorithm block of this module consists of a multi-head self-attention mechanism (MHSA) and a feedforward neural network (FFN), and the formula is expressed as:

[0075]

[0076] Where c is the text feature vector; yes The average value in the time dimension, that is, It is the prompt word for the video, which is the output of the prompt word module.

[0077] This module can use text feature vectors to extract visual descriptions related to the video. The original language description is enhanced to Where α is a learnable parameter. This is the output of the description word generator mentioned in the overall training framework section:

[0078]

[0079] Finally, each person's dynamic 2D video is scaled to a uniform size by frame and input into the above model to obtain the person's ongoing agricultural actions, and then automatically generate agricultural records. The next stage of agricultural production plan can be determined based on the current agricultural records. In this way, a system that can automatically detect and determine the type of agricultural actions is realized through a deep learning video action prediction model.

[0080] It is not difficult to find that in the embodiment of the present application, the use of deep learning models to extract features and detect actions of people can accurately identify the identities of all current personnel and the types of agricultural work they are performing, so as to carry out subsequent cultivation work more effectively and reasonably and ensure the rational development of agricultural operations.

[0081] Third embodiment

[0082] The third embodiment of the present application relates to a method for detecting farming types. The third implementation is an improvement on the first embodiment, and the specific improvements are:

[0083] The agricultural action recognition model includes: recognizing the character image in the dynamic 2D area to obtain a video clip and a text description, wherein the text description is a description of the agricultural action; inputting the video clip into the video encoder to obtain the video feature vector; inputting the text description into the text encoder to obtain the text feature vector; processing the video feature vector and the text feature vector through a video-oriented description word module to obtain an instance-level text representation of the corresponding video.

[0084] The similarity between the video feature vector and the instance-level text representation is calculated by using a cosine similarity function, and the action information of the operator is obtained according to the similarity.

[0085] The video encoder includes: a cross-frame communication converter and a multi-frame fusion converter; the cross-frame communication converter obtains the original frame of the video clip and uses a pre-trained language image model CLIP to provide a feature representation of the frame; the multi-frame fusion converter integrates the feature representation of each frame to obtain the video feature vector.

[0086] The text encoder includes: obtaining the text description according to the language description of the character image in the dynamic 2D area; the language description is a description of the farming category, including sowing, spraying pesticides and loosening the soil; processing the video feature vector and the text feature vector through a video-oriented prompt word module to obtain the enhanced text feature vector; the video-oriented prompt word module includes a multi-head self-attention algorithm and a feedforward neural network algorithm.

[0087] Based on the farming action records, determine the next stage of agricultural production plan.

[0088] It can be widely used in modern agricultural production management. Through video monitoring and deep learning technology, it can automatically identify the identity of operators and the agricultural actions performed, and provide agricultural managers with real-time and accurate operation records and data analysis support. In a large agricultural cooperative, managers need to monitor the agricultural operations in the farmland in real time to ensure that each operator performs tasks in accordance with regulations and can effectively evaluate and manage the operation results. Based on the agricultural type detection method of this embodiment, the cooperative can deploy a complete farmland operation monitoring system.

[0089] Cameras installed in various areas of the farmland can continuously capture farmland operation videos, especially during key farming stages such as sowing, irrigation, fertilization and harvesting. The cameras can cover multiple work areas and transmit videos to the central control system in real time.

[0090] When an operator enters the work area and starts to operate, the video analysis process is automatically started. First, the YOLOv3 model detects the operators in the video in real time and generates a dynamic 2D area for each operator to accurately track the position of each person. Then, the FaceNet model analyzes the operator's facial image and automatically identifies the identity information of each operator, such as work number or name.

[0091] For example, when a worker starts to plant rice in a rice field, the system can identify the worker as Zhang San through the FaceNet model and associate his identity information with the subsequent action recognition process.

[0092] After identifying the worker, the X-CL IP architecture is used to classify and identify the worker's actions. The video encoder extracts the worker's dynamic action features, and the text encoder converts the predefined agricultural action descriptions (such as "transplanting", "seeding", "fertilizing", etc.) into text features. By matching video features and text features, it automatically determines the agricultural operation currently being performed by the worker.

[0093] For example, Zhang San’s dynamic 2D area contains the action features of transplanting rice seedlings. After the system detects the action, it combines the text features to recognize that Zhang San is performing the “transplanting rice seedlings” operation.

[0094] Based on the above-identified worker identity and the actions performed, the system automatically generates a record of the operation task. The record includes the following information: worker: Zhang San; operation type: transplanting; operation time: 202x / x / x, 10:00 AM; operation duration: 2 hours. The record will be automatically stored in the database for subsequent management and analysis.

[0095] Based on the generated farming records, managers can conduct real-time evaluation of the completion of the current operation and formulate the next stage of agricultural production plan. The system automatically generates production suggestions and plans for the next stage based on the recorded information such as the identity of the operator, the type of farming action, the operation time and the progress of the operation, combined with the agricultural production cycle, climate conditions and crop growth.

[0096] Evaluate the current operation progress: Analyze the work progress of each operator in the record. For example, whether the sowing task of a certain batch has been completed, and whether the irrigation or fertilization operation has been carried out according to the scheduled time and standards. Predict crop growth needs: Based on the completed agricultural actions and crop growth stages, predict the needs of the next stage. For example, if the current record shows that fertilization has been completed, it will predict whether the next irrigation is needed based on the growth of the crop. Generate a production plan: Automatically generate the next stage of production tasks based on the identified operation completion status and predicted agricultural needs. For example, it is recommended to irrigate within the next three days and arrange the corresponding operators and equipment. Optimize resource allocation: Based on the generated plan, it can also help optimize resource allocation, such as adjusting personnel scheduling, equipment use and material procurement to ensure smooth agricultural production.

[0097] It is not difficult to find that in the embodiments of the present application, the agricultural type detection method can not only improve the transparency of farmland operations, but also reduce the errors and workload of manual recording, and significantly improve production efficiency. In addition, it can also be used to evaluate the work performance of operators to ensure that agricultural tasks are completed on time and accurately. The automated recording of agricultural data also helps to analyze and optimize the agricultural production process, ultimately improving agricultural output and economic benefits.

[0098] It should be noted that the third embodiment of the present application may also be an improvement based on the second embodiment.

[0099] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0100] Fourth embodiment

[0101] The fourth embodiment of the present application relates to a farming type detection system / device. Figure 3 As shown, the system includes:

[0102] A video processing module, used for processing the farmland video through a YOLOv3 model to obtain a dynamic 2D region of each operator;

[0103] An identity recognition module, used to recognize the person image in the dynamic 2D area through a FaceNet network model to obtain the identity information of the operator;

[0104] An action recognition module is used to recognize the human image in the dynamic 2D area through an agricultural action recognition model to obtain action information of the operator; the agricultural action recognition model adopts an X-CL IP architecture and includes a video encoder and a text encoder, the video encoder is used to extract a video feature vector, and the text encoder is used to extract a text feature vector;

[0105] The farming action recording module is used to obtain the farming action record according to the identity information and action information of the operator.

[0106] It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0107] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0108] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0109] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 4 An exemplary structural diagram of the electronic device is disclosed. Figure 4As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0110] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0111] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0112] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0113] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0114] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0115] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0116] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0117] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0118] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0120] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0121] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first", "second", etc. are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A method for detecting farming types, characterized in that: The method comprises: Acquire farmland videos, and process the farmland videos using the YOLOv3 model to obtain a dynamic 2D region of each operator; Recognize the person image in the dynamic 2D area through the FaceNet network model to obtain the identity information of the operator; The character image in the dynamic 2D area is recognized by a farming action recognition model to obtain the action information of the operator; the farming action recognition model adopts an X-CL IP architecture and includes a video encoder and a text encoder, the video encoder is used to extract a video feature vector, and the text encoder is used to extract a text feature vector; The agricultural action record is obtained according to the identity information and action information of the operator.

2. The method according to claim 1, characterized in that The FaceNet network model adopts a Zeiler&Fergus structure, and the FaceNet network model performs identity recognition on a person image by calculating the distance between feature vectors.

3. The method according to claim 1, characterized in that The farming action recognition model includes: Recognize the person image in the dynamic 2D area to obtain a video clip and a text description, wherein the text description is a description of the farming action; Input the video clip into the video encoder to obtain the video feature vector; input the text description into the text encoder to obtain the text feature vector; The video feature vector and the text feature vector are processed by a video-oriented description word module to obtain an instance-level text representation of the corresponding video.

4. The method according to claim 3, characterized in that The obtaining of the action information of the operator includes: The similarity between the video feature vector and the instance-level text representation is calculated by using a cosine similarity function, and the action information of the operator is obtained according to the similarity.

5. The method according to claim 3, characterized in that: The video encoder comprises: Cross-frame communication converter and multi-frame fusion converter; The cross-frame communication converter obtains the original frame of the video clip and provides a feature representation of the frame using a pre-trained language image model CLIP; The multi-frame fusion converter integrates the feature representations of each frame to obtain the video feature vector.

6. The method according to claim 3, characterized in that The text encoder comprises: According to the language description of the character image in the dynamic 2D area, the text description is obtained; the language description is a description of the farming category, including sowing, spraying pesticides and loosening the soil; The video feature vector and the text feature vector are processed by a video-oriented prompt word module to obtain an enhanced text feature vector; The video-oriented prompt word module includes a multi-head self-attention algorithm and a feedforward neural network algorithm.

7. The method according to any one of claims 1 to 6, characterized in that: The method also includes: judging the agricultural production plan for the next stage according to the farming action records.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 7.

9. A computer readable medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images