A Deep Learning-Based Method and Device for Agricultural Machinery Trajectory Recognition

Abstract

This application provides a method and apparatus for agricultural machinery trajectory recognition based on deep learning. The method includes: acquiring multiple trajectory points, each trajectory point including position information, speed information, time information, and mileage information; determining a first three-channel trajectory map based on the multiple trajectory points, the first three-channel trajectory map including a speed map, a time map, and a mileage map. The speed map includes the speed information of the multiple trajectory points, the time map includes the time information of the multiple trajectory points, and the mileage map includes the mileage information of the multiple trajectory points. A pre-trained neural network model is used to process the first three-channel trajectory map to determine the types of different trajectory segments in the first three-channel trajectory map. The solution provided by this application can improve the accuracy of agricultural machinery trajectory type recognition.

Description

Technical Field

[0001] This application relates to the agricultural field, specifically to a method and apparatus for recognizing agricultural machinery trajectories based on deep learning. Background Technology

[0002] With the development of technology, agricultural mechanization is becoming increasingly intelligent. Among these, the identification of agricultural machinery operation trajectories plays a crucial role in statistical analysis of operation area, research on crop distribution, and planning operation routes.

[0003] In existing technologies, it is difficult to accurately distinguish between roads and farmland based on the operation trajectory of agricultural machinery. Therefore, how to provide a more accurate method for identifying agricultural machinery trajectories is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a method and apparatus for identifying agricultural machinery trajectories based on deep learning, which can accurately distinguish between roads and farmland from trajectory maps, thereby improving the accuracy of agricultural machinery trajectory type identification.

[0005] Firstly, this application provides a deep learning-based method for agricultural machinery trajectory recognition. The method provided in this application includes: acquiring multiple trajectory points, each trajectory point including position information, speed information, time information, and mileage information; determining a first three-channel trajectory map based on the multiple trajectory points, the first three-channel trajectory map including a speed map, a time map, and a mileage map, the speed map including speed information of the multiple trajectory points, the time map including time information of the multiple trajectory points, and the mileage map including mileage information of the multiple trajectory points; and processing the first three-channel trajectory map using a pre-trained neural network model to determine the types of different trajectory segments in the first three-channel trajectory map.

[0006] The deep learning-based agricultural machinery trajectory recognition method provided in this embodiment defines the value corresponding to each channel in the three-channel trajectory map as the value corresponding to speed, time, and mileage, thereby obtaining speed, time, and mileage maps for multiple trajectory points. Through analysis and processing by a neural network model, the types of different trajectory segments in the three-channel trajectory map can be determined based on the features of the position, speed, time, and mileage information in the map. This method can improve the accuracy of agricultural machinery trajectory type recognition.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the first three-channel trajectory map includes different trajectory segments and a background. The neural network model includes a post-processing layer, which determines the background through matrix operations on a first vector and determines the type of different trajectory segments through matrix operations on a second vector. The first and second vectors are binary vectors of length 3. The post-processing layer is a Softmax layer.

[0008] In the above scheme, by setting a post-processing layer in the neural network model, matrix operations of different vectors can be used to distinguish the trajectory segments and background in the three-channel trajectory map, which can avoid the boundary blurring and scattered point problems of trajectory recognition, thereby improving the accuracy of trajectory recognition.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes training a neural network model to obtain a pre-trained neural network model. Training the neural network model includes acquiring a second or third-channel trajectory map, and training the neural network model using the labeled second or third-channel trajectory map, wherein the labels correspond to different trajectory segment types in the second or third-channel trajectory map. It should be understood that different trajectory segment types can be distinguished by labels of different colors or symbols.

[0010] It should be understood that the second and third channel trajectory diagrams in this application embodiment include a velocity diagram, a time diagram, and a mileage diagram of the trajectory. The velocity diagram, time diagram, and mileage diagram are grayscale diagrams, and the magnitude of the grayscale value in the grayscale diagram corresponds to the velocity magnitude, time magnitude, or mileage magnitude of the trajectory.

[0011] In the above scheme, by learning the features and corresponding labels in the three-channel trajectory map, a neural network model of the three-channel trajectory map can be established. This neural network model can learn the characteristics of the trajectory in terms of speed, time and distance, thereby improving the accuracy of trajectory recognition.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, training the neural network model further includes modifying the neural network model using a loss function, wherein the loss function satisfies FL(p t )=-α t (1-p t )γlog(p t ), where p t p represents the probability that the predicted sample belongs to 1. t ∈[0,1], γ is the adjustment factor, γ>0, α t ∈[2.5,3].

[0013] It should be understood that the loss function in the embodiments of this application is the Focal Loss function. Since the distribution of road segment trajectory and farmland segment trajectory in agricultural machinery trajectory is uneven, and the farmland segment trajectory is much larger than the road segment trajectory, in order to avoid the model biased towards farmland, this application uses the loss function to correct the neural network model in the above scheme, thereby improving the trajectory recognition accuracy.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the different types of trajectory segments in the first three-channel trajectory map include road segment trajectories and farmland segment trajectories. The farmland segment trajectory includes a work area and a U-turn area. The method in this application embodiment further includes determining the work area based on the U-turn area, where the work area is the area enclosed by the U-turn area.

[0015] In the above scheme, the farmland section trajectory can be divided into operation area and turning area based on trajectory characteristics. The operation area can be determined based on the turning area, and the size of the operation area can be accurately determined, which is convenient for statistics and research.

[0016] In conjunction with the first aspect, in certain implementations of the first aspect, the different types of trajectory segments in the first three-channel trajectory map include error trajectory segments, where the velocity difference between the error trajectory segment and the trajectory segment at an adjacent time is greater than a first threshold, and / or, the mileage difference between the error trajectory segment and the trajectory segment at an adjacent time is greater than a second threshold. The neural network model is also used to exclude error trajectory segments from the first three-channel trajectory map.

[0017] It should be understood that there will be error trajectory points among multiple trajectory points. This is because in some areas, high-voltage power poles or other signal sources interfere with GPS, causing errors in the position, speed, time, and mileage information of these error trajectory points. By learning the characteristics of error trajectory points, the neural network model can eliminate multiple error trajectory points from multiple trajectory points, thereby improving the accuracy of trajectory recognition.

[0018] Secondly, this application provides an agricultural machinery trajectory recognition device. The agricultural machinery trajectory recognition device provided in this application includes a processing unit and a storage unit. The storage unit is used to store a neural network model. The processing unit is used to acquire multiple trajectory points, each of which includes position information, speed information, time information, and mileage information. A first three-channel trajectory map is determined based on the multiple trajectory points. The first three-channel trajectory map includes a speed map, a time map, and a mileage map. The speed map includes the speed information of the multiple trajectory points, the time map includes the time information of the multiple trajectory points, and the mileage map includes the information of the multiple trajectory points. The neural network model is used to process the first three-channel trajectory map to determine the types of different trajectory segments in the first three-channel trajectory map.

[0019] In conjunction with the second aspect, in some implementations of the second aspect, the first three-channel trajectory map includes different trajectory segments and a background. The neural network model includes a post-processing layer, which determines the background through matrix operations on a first vector and determines the type of different trajectory segments through matrix operations on a second vector. The first and second vectors are binary vectors of length 3.

[0020] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is further configured to train a neural network model to obtain a pre-trained neural network model. Training the neural network model includes acquiring a second / third-channel trajectory map. The neural network model is trained using the labeled second / third-channel trajectory map, wherein the labels correspond to different trajectory segment types in the second / third-channel trajectory map.

[0021] In conjunction with the second aspect, some implementations of the second aspect also include training the neural network model by using a loss function to modify the neural network model, where the loss function satisfies FL(p t )=-α t (1-p t ) γ log(p t ), where p t p represents the probability that the predicted sample belongs to 1. t ∈[0,1], γ is the adjustment factor, γ>0, α t ∈[2.5,3].

[0022] In conjunction with the second aspect, in some implementations of the second aspect, the different types of trajectory segments in the first three-channel trajectory map include road segment trajectories and farmland segment trajectories, with the farmland segment trajectory including the work area and the U-turn area. The processing unit is also used to determine the work area based on the U-turn area, where the work area is the area enclosed by the U-turn area.

[0023] In conjunction with the second aspect, in some implementations of the second aspect, the different types of trajectory segments in the first three-channel trajectory map include error trajectory segments, where the velocity difference between the error trajectory segment and the trajectory segment at an adjacent time is greater than a first threshold, and / or, the mileage difference between the error trajectory segment and the trajectory segment at an adjacent time is greater than a second threshold. The processing unit is further configured to: exclude error trajectory segments from the first three-channel trajectory map.

[0024] Thirdly, this application provides a computer-readable medium storing instructions for execution by a device, the instructions being used to implement the method in the first aspect or any of the possible implementations thereof.

[0025] Fourthly, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method in the first aspect or any of the possible implementations thereof.

[0026] Fifthly, this application provides a chip, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the method in the first aspect or any of the possible implementations thereof.

[0027] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform the method in the first aspect or any of the possible implementations thereof.

[0028] In a sixth aspect, this application provides a computing device, which includes a processor and a memory, wherein: the memory stores computer instructions, and the processor executes the computer instructions to implement the method in the first aspect or any of the possible implementations thereof. Attached Figure Description

[0029] Figure 1 This is a schematic flowchart of a deep learning-based agricultural machinery trajectory recognition method provided in an embodiment of this application.

[0030] Figure 2 This is a schematic diagram of the distribution of trajectory points provided in an embodiment of this application.

[0031] Figure 3 This is a schematic diagram of a three-channel trajectory diagram provided in an embodiment of this application.

[0032] Figure 4 This application provides a three-channel trajectory map corresponding to a velocity map, a time map, and a mileage map.

[0033] Figure 5 This is a comparative schematic diagram of time graphs corresponding to different three-channel trajectory graphs provided in the embodiments of this application.

[0034] Figure 6 This is a velocity map corresponding to a three-channel trajectory map provided in an embodiment of this application.

[0035] Figure 7 This is a velocity map corresponding to another three-channel trajectory map provided in this application embodiment.

[0036] Figure 8 This is a schematic diagram of the structure of a neural network model provided in an embodiment of this application.

[0037] Figure 9 This is a schematic diagram of a model training process provided in an embodiment of this application.

[0038] Figure 10 This is a schematic block diagram of an image super-resolution reconstruction apparatus according to an embodiment of this application.

[0039] Figure 11 This is a schematic block diagram of an image super-resolution reconstruction apparatus according to another embodiment of this application. Detailed Implementation

[0040] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0041] In the description of the embodiments in this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0042] The terms "first," "second," "third," "fourth," etc., as used in this application (if applicable), are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The sequence numbers of the processes below do not imply a specific order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. For example, in the embodiments of this application, the terms "110," "210," "220," etc., are merely identifiers for descriptive convenience and are not intended to limit the system.

[0043] References to “some embodiments” and the like in this specification mean that one or more embodiments of this application include a particular feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as “some embodiments” appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean “one or more, but not all, embodiments”, unless otherwise specifically emphasized. The terms “comprising,” “including,” “having,” and variations thereof mean “including, but not limited to,” unless otherwise specifically emphasized.

[0044] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms “a” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0045] First, let's introduce the application scenarios of this application.

[0046] In recent years, with the development of science and technology, agricultural mechanization has become increasingly intelligent. Among these, the identification of agricultural machinery operation trajectories plays a crucial role in statistical analysis of operation area, research on crop distribution, and planning operation routes.

[0047] In the identification of agricultural machinery operation trajectories, since the trajectory patterns of road sections and farmland sections are often very similar, traditional methods for identifying agricultural machinery operation trajectories are difficult to accurately distinguish between them.

[0048] In view of this, embodiments of this application provide a method and apparatus for agricultural machinery trajectory recognition based on deep learning, which can accurately distinguish between roads and farmland from trajectory maps, thereby improving the recognition accuracy of agricultural machinery trajectory types.

[0049] To make the technical problems solved by this application, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0050] Furthermore, the methods and apparatus described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. Those skilled in the art will know that the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0051] It should also be understood that the various implementation methods described in this specification can be implemented individually or in combination, and the embodiments of this application are not limited in this respect.

[0052] To facilitate understanding, we will first provide a detailed introduction to neural networks. A neural network typically consists of multiple layers, each performing different operations or calculations. Common neural network layers include convolutional layers and pooling layers. Multiple convolutional layers and multiple pooling layers are arranged alternately; a convolutional layer can be followed by either another convolutional layer or a pooling layer.

[0053] In some embodiments, the neural network model in this application refers to a convolutional neural network (CNN).

[0054] Convolutional layers are primarily used to perform convolution operations on input matrices, while pooling layers are primarily used to perform pooling operations on input matrices. Both convolution and pooling operations correspond to a kernel; the kernel for a convolution operation is called a convolution kernel. The following sections describe convolution and pooling operations in detail.

[0055] Convolution operations are primarily used in image processing, where the input matrix is ​​also known as a feature map. Each convolution operation corresponds to a convolution kernel, also called a weight matrix, where each element represents a weight. During convolution, the input matrix is ​​divided into many submatrices of the same size as the weight matrix by a sliding window. Each submatrix is ​​multiplied by the weight matrix, and the result is a weighted average of the data elements in each submatrix.

[0056] Pooling operations are generally used to reduce the dimensionality of the input matrix, i.e., downsampling the input matrix. Similar to convolution, pooling also uses a kernel to perform calculations on the input matrix; therefore, it also uses a sliding window, and the stride of the pooling operation is usually greater than 1 (but can also be equal to 1). There are several types of pooling operations, such as average pooling and max pooling. Average pooling averages all elements in the sliding window. Max pooling calculates the maximum value among all elements in the sliding window. The pooling process is largely similar to the convolution process, the difference being the different operations performed on the data elements in the sliding window, which will not be detailed here.

[0057] Figure 1 This is a schematic flowchart illustrating a deep learning-based agricultural machinery trajectory recognition method provided in an embodiment of this application. Figure 1 As shown, the method 100 includes:

[0058] S110, acquire multiple trajectory points, each of which includes location information, speed information, time information, and distance information.

[0059] Agricultural machinery refers to mechanized equipment used in agricultural production, such as tractors, combine harvesters, tillers, and plant protection machines. Multiple trajectory points can be uploaded by the Global Positioning System (GPS) installed on the agricultural machinery. For example, GPS uploads a trajectory point at regular intervals, which includes location, speed, time, and mileage information.

[0060] Specifically, the location information includes the latitude and longitude of the agricultural machinery at the track point, the speed information includes the speed of the agricultural machinery at the track point, the time information includes the cumulative time from the first track point to the current track point, and the mileage information includes the cumulative mileage from the first track point to the current track point.

[0061] Figure 2 This is a schematic diagram of the distribution of trajectory points provided in an embodiment of this application. For example... Figure 2 As shown, a distribution map of trajectory points can be drawn based on the location information of multiple trajectory points.

[0062] S120, determine the first three-channel trajectory map based on multiple trajectory points. The first three-channel trajectory map includes a speed map, a time map, and an odometer map. The speed map includes speed information of multiple trajectory points, the time map includes time information of multiple trajectory points, and the odometer map includes odometer information of multiple trajectory points.

[0063] Specifically, firstly, the multiple trajectory points obtained in step S110 are mapped to pixels on a map based on the location information of each trajectory point. The location information includes latitude and longitude, and the map can be a satellite image or a background image. Then, the multiple trajectory points are sequentially connected on the map according to their time sequence, thereby creating a first three-channel trajectory map. It should be understood that everything outside the trajectories in the three-channel trajectory map is background; the background area contains no speed, time, or distance information, meaning the values ​​of speed, time, and distance information in the background area are all 0.

[0064] Figure 3 This is a schematic diagram of a three-channel trajectory diagram provided in an embodiment of this application. Figure 3 As shown, based on the location information, speed information, time information, and mileage information of multiple trajectory points, the first and third channel trajectory diagrams corresponding to multiple trajectory points can be drawn.

[0065] It should be understood that common color images are composed of three channels—red, green, and blue (RGB). In an RGB image, the value corresponding to each channel represents the brightness value under red, green, and blue light. By combining these channels appropriately, a color image can be generated, where the intensity and saturation of each color can be adjusted by distorting the channel values. Semantic segmentation can assign each pixel in an RGB image to a specific category.

[0066] In this embodiment, the three-channel trajectory map uses custom data to define the three channels of the trajectory map as speed, time, and distance. That is, the first three-channel trajectory map simultaneously includes information on speed, cumulative time, and cumulative distance, and is composed of three channels—speed, time, and distance (STD). Thus, by extracting information from different channels, namely the speed, time, and distance information of the corresponding trajectory, the speed map, time map, and distance map corresponding to the first three-channel trajectory map can be obtained. For example, by extracting the speed information from the speed channel of the first three-channel trajectory map, the speed map corresponding to the first three-channel trajectory map can be obtained. In this speed map, the trajectory contains speed information for multiple trajectory points, and the speed information value of the background area is 0.

[0067] In this embodiment, the first three-channel trajectory map determined based on multiple trajectory points can be selected with different sizes or map levels. For example, to avoid losing details due to a map level that is too low or failing to obtain more global information due to a map level that is too high, an 18-level satellite image can be selected; to avoid the image being too large and reducing the model's running speed, the image size can be selected as 1024×1024.

[0068] In some embodiments, to ensure that the model can extract sufficiently rich edge information, or to meet the size requirements of the model input, an overlap technique is used to add edge information and expand the original image. For example, adding 128 pixels of edge information to the edges of an image with a size of 1024×1024 expands the image to a size of 1280×1280.

[0069] S130, using a pre-trained neural network model to process the first three-channel trajectory map to determine the types of different trajectory segments in the first three-channel trajectory map.

[0070] In some embodiments, the trajectory segments are divided by time or distance. For example, a trajectory segment may be defined as every 10 minutes or every 10 meters.

[0071] By inputting the first three-channel trajectory map into a pre-trained neural network model, the different types of trajectory segments in the first three-channel trajectory map can be obtained. These different trajectory segment types can be distinguished by color or labeling. For example, road segments are represented by red, and farmland segments by green, respectively.

[0072] In this way, by defining the value corresponding to each channel in the three-channel trajectory map as the value corresponding to speed, time, and mileage, speed, time, and mileage maps of multiple trajectory points can be obtained. Furthermore, through analysis and processing by a neural network model, the types of different trajectory segments in the three-channel trajectory map can be determined based on the characteristics of the position, speed, time, and mileage information in the map. This method can improve the accuracy of agricultural machinery trajectory type recognition.

[0073] In this embodiment, the model input is a custom three-channel trajectory map, which is composed of three channels—velocity, time, and distance (STD). Specifically, the velocity map is determined based on the velocity channel input, the time map is determined based on the time channel input, and the distance map is determined based on the distance channel input. In particular, grayscale values ​​represent numerical values ​​in the images corresponding to different channels.

[0074] For example, Figure 4 This application provides a three-channel trajectory map corresponding to a velocity map, a time map, and an odometer map. For example... Figure 4 As shown, Figure 4(a) in the figure is a velocity graph, in which the brighter the color of the trajectory segment, the higher the gray value, indicating that the velocity on that trajectory segment is faster; the darker the color of the trajectory segment, the lower the gray value, indicating that the velocity on that trajectory segment is slower. Figure 5 (b) in the diagram is a time graph, where the brighter the color of the trajectory segment, the higher the gray value, indicating that the agricultural machinery has been working for a longer period of time on that trajectory segment. Figure 4 (c) in the diagram represents the mileage map, where the brighter the color of the trajectory segment, the higher the gray value, indicating that the agricultural machinery has accumulated a longer mileage on that trajectory segment.

[0075] It should be understood that, by inputting a custom three-channel trajectory graph, the embodiments of this application can extract features of different trajectory segment types based on the speed, time, and mileage information in the three-channel trajectory graph. The following description provides examples of some embodiments.

[0076] In some embodiments, the different types of trajectory segments in the first three-channel trajectory map include road segment trajectories and farmland segment trajectories.

[0077] Figure 5 This is a time map corresponding to a different three-channel trajectory map provided in an embodiment of this application. For example... Figure 5 As shown, Figure 5 (a) and Figure 5 In (b), each includes two relatively parallel trajectories. Figure 5 (a) includes trajectory segment 1 and trajectory segment 2, where the gray value of trajectory segment 1 is much smaller than that of trajectory segment 2. Specifically, the formation time of trajectory segment 1 can be determined to be earlier than that of trajectory segment 2 by 10 hours based on the magnitude of the gray value. Figure 5 (b) includes trajectory segment 3 and trajectory segment 4, where the grayscale value of trajectory segment 3 is close to that of trajectory segment 4, indicating that trajectory segment 3 and trajectory segment 4 were formed at similar times. Therefore, it can be determined that... Figure 5 In (a), trajectory segments 1 and 2 are road segment trajectories. Figure 5 In (b), trajectory segments 3 and 4 are farmland segments.

[0078] In some embodiments, the farmland section trajectory includes a work area and a turning area. The method in this application embodiment further includes determining the work area based on the turning area, where the work area is the area enclosed by the turning area.

[0079] Figure 6 This is a velocity map corresponding to a three-channel trajectory map provided in an embodiment of this application. For example... Figure 6 As shown, Figure 6The farmland section trajectory includes U-turn area 1, U-turn area 2, and the work area. The grayscale values ​​of the trajectory segments in U-turn area 1 and U-turn area 2 are lower than those in the work area trajectory segment, meaning the speed of the work area trajectory segment is greater than the speed of the U-turn area trajectory segment. Therefore, the farmland section trajectory can be further divided into the work area and the U-turn area. Specifically, the work area is the area enclosed by the U-turn area.

[0080] In some embodiments, the different types of trajectory segments in the first three-channel trajectory map include error trajectory segments, where the velocity difference between the error trajectory segment and a trajectory segment at an adjacent time is greater than a first threshold, and / or, the mileage difference between the error trajectory segment and a trajectory segment at an adjacent time is greater than a second threshold. The neural network model is also used to exclude error trajectory segments from the first three-channel trajectory map.

[0081] Figure 7 This is another velocity map corresponding to a three-channel trajectory map provided in an embodiment of this application. For example... Figure 7 As shown, Figure 7 One end of the work area contains a bomb crater. It should be understood that in certain areas of farmland, high-voltage power poles or other signal sources may interfere with GPS, causing errors in the uploaded trajectory points. The trajectory segments determined based on these erroneous trajectory points are called error trajectory segments. Specifically, an error trajectory segment is defined as one whose velocity difference from an adjacent time trajectory segment is greater than a first threshold, and / or whose mileage difference from an adjacent time trajectory segment is greater than a second threshold. For example, if the first threshold is 5, and the velocity of a farmland section trajectory during normal cultivation is 10 meters per second, then if the velocity of an adjacent time trajectory segment is greater than 15 meters per second or less than 5 meters per second, this trajectory segment is determined to be an error trajectory segment and excluded from the first three-channel trajectory map. The method for determining error trajectory segments based on mileage differences is similar to the method described above and will not be repeated here.

[0082] In this embodiment, the first three-channel trajectory map includes different trajectory segments and a background. The neural network model includes a post-processing layer, which determines the background through matrix operations on a first vector and determines the type of different trajectory segments through matrix operations on a second vector. The first and second vectors are binary vectors of length 3. Specifically, the post-processing layer is a Softmax layer.

[0083] Specifically, in this embodiment, the input to the neural network model is a three-channel trajectory map determined based on multiple trajectory points, and the output of the neural network model is the different types of trajectory segments in the aforementioned three-channel trajectory map. In other words, after the three-channel trajectory map is input into the neural network model, it is processed by the neural network model, and finally outputs the different types of trajectory segments in the three-channel trajectory map.

[0084] It should be understood that in this embodiment, the neural network model may be pre-trained. In this case, in this embodiment, the method further includes training the neural network model to obtain a pre-trained neural network model. Training the neural network model includes obtaining a second three-channel trajectory map, and training the neural network model using the labeled second three-channel trajectory map, wherein the labels correspond to different trajectory segment types in the second three-channel trajectory map. For example, different trajectory segment types can be distinguished by different colors or different symbols.

[0085] Specifically, the neural network model first acquires pre-labeled three-channel trajectory maps. For example, representative three-channel trajectory maps can be selected, and their different trajectory segment types can be manually labeled as training samples. It should be understood that after training the neural network model, in subsequent image trajectory recognition, the neural network model can be directly used to determine the types of different trajectory segments in the three-channel trajectory map without needing to train the neural network model again.

[0086] The following describes in detail, with specific examples, the method for training a neural network model and the method for using the neural network model to determine the types of different trajectory segments in a three-channel trajectory graph according to the embodiments of this application.

[0087] The neural network model in this application embodiment can be constructed based on a deep learning semantic segmentation algorithm. Specifically, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a neural network model provided in an embodiment of this application. The structure of this neural network model may include an input layer, an encoding layer, an information interaction layer, a connection layer, a decoding layer, and an output layer. The parameters of each of the above layers are described in detail below.

[0088] The first input layer contains two consecutive convolutional layers. Convolutional layer 1 contains a 3×3 convolutional kernel with a stride of 2 and 12 channels. Convolutional layer 2 contains a 3×3 convolutional kernel with a stride of 2 and 24 channels.

[0089] For example, suppose the three-channel trajectory map to be identified is an image with a size of 1280×1280×3, and the standard size of the divided image blocks is 4×4. Since the size of the three-channel trajectory map to be identified is an integer multiple of the image block size, the three-channel trajectory map to be identified can be divided into multiple 4×4 image blocks, and the 3-channel STD can be expanded into 24 channels.

[0090] The second coding layer consists of four stages, each composed of multiple blocks. Each block uses a deep-wise convolutional structure. It should be understood that deep-wise convolutions can independently convolve each channel of the input data, thus extracting image features more effectively.

[0091] Specifically, such as Figure 8 As shown, stage 1 uses 2 blocks with 24 input channels and 32 output channels; stage 2 uses 2 blocks with 32 input channels and 64 output channels; stage 3 uses 6 blocks with 64 input channels and 128 output channels; and stage 4 uses 2 blocks with 128 input channels and 256 output channels. That is, feature maps are extracted and output at downsampling scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively.

[0092] For example, after the image is downsampled by 32 times in the second coding layer, the size of the feature map changes from the original 1280×1280×3 to 40×40×256 (denoted as X), where 40×40 represents the height and width of the feature map, and each point corresponds to 32×32 points in the original image. To further extract global information, the third information interaction layer is divided into three branches for pooling and convolution, and finally merged together.

[0093] Specifically, in the first branch, the pooling layer has a convolution kernel size of 2 and a stride of 2, resulting in a feature map size of 20×20×256; the convolution layer has a convolution kernel size of 5 and a stride of 1, with 256 channels. Using a convolution kernel of size 5 allows for the utilization of 5 / 20 = 25% of the global information.

[0094] In the second branch, the pooling layer has a convolution kernel size of 4 and a stride of 4, resulting in a feature map size of 10×10×256. The convolutional layer has a convolution kernel size of 5, a stride of 1, and 256 channels. Using a convolution kernel of size 5 allows us to utilize 5 / 10 = 50% of the global information.

[0095] In the third branch, the pooling layer has a convolution kernel size of 8 and a stride of 8, resulting in a feature map size of 5×5×256. The convolutional layer has a convolution kernel size of 5, a stride of 1, and 256 channels. Using a convolution kernel of size 5 allows us to utilize 5 / 5 = 100% of the global information.

[0096] After three pooling convolution operations along the three branches, three feature maps of 5×5×256, 10×10×256, and 20×20×256 are obtained. These three feature maps are then upsampled to a size of 40×40×256, and concatenated together to form a 40×40×768 feature map, where 768 = 3×256, representing the number of channels in the three feature maps. A further convolutional layer restores the feature count to 40×40×256 and adds it to the input X, completing the short-cut operation.

[0097] The fourth connection layer directly connects the four feature maps extracted from the four stages of the second encoding layer with the corresponding four stages of the fifth decoding layer. That is, it directly maps the raw information from the four stages of the encoding layer to the corresponding levels of the decoding layer.

[0098] The fifth decoding layer can restore the information extracted from the second encoding layer to the original resolution scale layer by layer, and fuse the features of various channels to transform them into classification features.

[0099] The sixth output layer further transforms the feature map into a classification result. In this embodiment, the output of the three-channel trajectory map can be divided into three categories: background, road segment trajectory, and farmland segment trajectory. Specifically, a convolutional layer with three channels is used to map the feature map onto the three categories, then a function operation is used to obtain the probability value of each category, and finally the category with the highest probability is taken as the output result.

[0100] In this embodiment, the three-channel trajectory map includes different trajectory segments and a background. The neural network model includes a seventh post-processing layer. The post-processing layer determines the background through matrix operations on a first vector and determines the type of different trajectory segments through matrix operations on a second vector. The first and second vectors are binary vectors of length 3.

[0101] For example, normalization is performed using the softmax function. The model outputs three classification probability values ​​[p1, p2, p3] at each point after softmax, representing the probabilities of [background, road segment trajectory, and farmland segment trajectory]. Specifically, the first vector is [1, 0, 0]. Multiplying [1, 0, 0] by [p1, p2, p3] yields [p1, 0, 0], representing the classification result of the background region. In this case, since the probabilities of road segment and farmland segment trajectories are both set to zero, the classification result for these regions can only be background. The second vector is [0, 1, 1]. Multiplying [0, 1, 1] by [p1, p2, p3] yields [0, p2, p3]. Since the probability of background is also set to zero, the classification result for the trajectory can only be either a road segment trajectory or a farmland segment trajectory.

[0102] It should be noted that the above only lists one feasible scheme for classifying trajectory segments using a neural network model, but the embodiments of this application are not limited thereto. In practical applications, the above classification model can be modified in certain ways, such as increasing or decreasing the number of convolutional layers and pooling layers, or increasing or decreasing the number of convolutional kernels in each convolutional layer. Such modifications or changes also fall within the scope of the embodiments of this application.

[0103] The specific training process is described below:

[0104] First, representative three-channel trajectory maps are selected and their types are manually labeled as training samples. Let's assume trajectory segment x in the three-channel trajectory map... i The category is labeled y i There are a total of m training samples, i.e., {(x 1 ,y 1 ),(x 2 ,y 2 ),…,(x m ,y m )}.

[0105] Input sample set {(x) 1 ,y 1 ),(x 2 ,y 2 ),…,(x m ,y m Each sample in the array is sequentially input into the constructed neural network, the network parameters are randomly initialized, and the initial actual output is calculated.

[0106] Then, in this embodiment, a loss function can be used to modify the neural network model to obtain the parameters of the trained neural network model. Specifically, the loss function satisfies FL(p t )=-α t (1-p t ) γ log(p t ), where p t p represents the probability that the predicted sample belongs to 1. t ∈[0,1], γ is the adjustment factor, γ>0, α t For sample weights, α t ∈[2.5,3].

[0107] It should be understood that the loss function in the embodiments of this application is the Focal Loss function. Since the distribution of road segment trajectory and farmland segment trajectory in agricultural machinery trajectory is uneven, and the farmland segment trajectory is much larger than the road segment trajectory, in order to avoid the model biased towards farmland, this application uses the loss function to correct the neural network model in the above scheme, thereby improving the trajectory recognition accuracy.

[0108] It should be understood that the method for training a classification model described above is merely an example, and various variations can be made in practical applications, all of which fall within the scope of the embodiments of this application. For example, in actual training, the parameters in the pooling layer can also be trained, and the embodiments of this application do not limit this.

[0109] Figure 9 This is a schematic diagram illustrating a model training process provided in an embodiment of this application. During training, each data batch size is set to 8, for a total of 48 batches. During the training of each batch, a learning rate is used to control the rate of model parameter updates, wherein the learning rate is set to 5e. -3 .like Figure 9 As shown, in order to improve the accuracy of the model, a warm-up strategy is adopted, such as a warm-up operation. The model is first trained with a low learning rate, and then gradually reduced after the model converges to a certain extent and reaches the maximum learning rate. This helps to avoid the problem of decreased accuracy caused by the model updating too quickly.

[0110] In the above text, combined with Figures 1 to 9 The deep learning-based agricultural machinery trajectory recognition method of this application embodiment is described in detail below, in conjunction with... Figure 10 and Figure 11 This application describes an agricultural machinery trajectory recognition device according to an embodiment of the present application.

[0111] Figure 10 This is a schematic block diagram of an agricultural machinery trajectory recognition device according to an embodiment of this application. It should be understood that the agricultural machinery trajectory recognition device (hereinafter referred to as the "recognition device") in this application embodiment can be any device with agricultural machinery trajectory recognition function in various fields requiring agricultural machinery trajectory recognition, such as satellite remote sensing image analysis and surveying and mapping. This includes, but is not limited to, cloud service or cloud computing devices (e.g., cloud service platforms), satellite remote sensing image analysis devices, and surveying and mapping devices. This application embodiment is not limited to these; any recognition device that can implement the method of agricultural machinery trajectory recognition is acceptable. Optionally, the recognition device can also be a chip.

[0112] Figure 10 The agricultural machinery trajectory recognition device 1000 shown includes a processing unit 1010 and a storage unit 1020.

[0113] Specifically, the storage is used to store a neural network model, and the processing unit is used to: acquire multiple trajectory points, each of which includes position information, speed information, time information, and odometer information; determine a first three-channel trajectory map based on the multiple trajectory points, the first three-channel trajectory map including a speed map, a time map, and an odometer map. The speed map includes the speed information of the multiple trajectory points, the time map includes the time information of the multiple trajectory points, and the odometer map includes the information of the multiple trajectory points; process the first three-channel trajectory map using the neural network model to determine the types of different trajectory segments in the first three-channel trajectory map.

[0114] Therefore, in this embodiment, the features corresponding to speed, time and mileage in the three-channel trajectory map are extracted by a neural network model, thereby determining the types of different trajectory segments in the three-channel trajectory map, which can improve the recognition accuracy of agricultural machinery trajectory types.

[0115] Optionally, as another embodiment, the first three-channel trajectory map includes different trajectory segments and a background. The neural network model includes a post-processing layer, which determines the background through matrix operations on a first vector and determines the type of different trajectory segments through matrix operations on a second vector, wherein the first and second vectors are binary vectors of length 3.

[0116] Optionally, as another embodiment, the processing unit is further configured to train the neural network model to obtain a pre-trained neural network model. Training the neural network model includes acquiring a second three-channel trajectory map. The neural network model is trained using the labeled second three-channel trajectory map, wherein the labels correspond to different trajectory segment types in the second three-channel trajectory map.

[0117] Optionally, as another embodiment, training the neural network model further includes modifying the neural network model using a loss function, wherein the loss function satisfies FL(p t )=-α t (1-p t ) γ log(p t ), where p t p represents the probability that the predicted sample belongs to 1. t ∈[0,1], γ is the adjustment factor, γ>0, α t ∈[2.5,3].

[0118] Optionally, as another embodiment, the different types of trajectory segments in the first three-channel trajectory map include road segment trajectories and farmland segment trajectories, with the farmland segment trajectory including a work area and a U-turn area. The processing unit is further configured to determine the work area based on the U-turn area, wherein the work area is the area enclosed by the U-turn area.

[0119] Optionally, as another embodiment, the different types of trajectory segments in the first three-channel trajectory map include error trajectory segments, where the velocity difference between the error trajectory segment and the trajectory segment at an adjacent time is greater than a first threshold, and / or, the mileage difference between the error trajectory segment and the trajectory segment at an adjacent time is greater than a second threshold. The processing unit is further configured to: exclude error trajectory segments from the first three-channel trajectory map.

[0120] Optionally, Figure 10 The identification device may also include a transceiver unit, which may also be called a communication unit. For example, it may be a network interface. The processing unit can obtain the image to be identified or obtain a three-channel trajectory map sample through the transceiver unit. The embodiments of this application are not limited thereto.

[0121] It should be understood that Figure 10 The identification device shown can achieve Figures 1 to 9 The various processes in the method embodiments, the operations and / or functions of the various modules in the identification device 1000, are respectively for the purpose of implementing Figures 1 to 9 The corresponding processes in the method embodiments are detailed in the descriptions in the above method embodiments. To avoid repetition, detailed descriptions are omitted here.

[0122] Figure 11 A schematic block diagram of an agricultural machinery trajectory recognition device 1100 according to an embodiment of this application is shown. Specifically, as Figure 11 As shown, the agricultural machinery trajectory recognition device 1100 includes a processor 1110 and a memory 1120, which are connected together. Optionally, the agricultural machinery trajectory recognition device 1100 also includes a transceiver 1130, which is connected to the processor 1110. The processor 1110, the transceiver 1130, and the memory 1120 communicate with each other through an internal connection path to transmit control and / or data signals.

[0123] The transceiver 1130 can be an input unit, a communication unit, a network interface, or other components. The processor 1110 can acquire the image to be recognized or acquire a three-channel trajectory map sample through the transceiver 1130. The memory 1120 can be used to store data, such as classification models and recognition models. The memory 1120 can also be used to store instructions. The processor 1110 executes the instructions stored in the memory 1120 to control the transceiver 1120 to send and receive information or signals. The controller 1110 can perform the above-mentioned tasks by executing the instructions in the memory 1120. Figures 1 to 9 The various processes in the method embodiments are described below. To avoid repetition, they will not be repeated here.

[0124] It should be understood that the agricultural machinery trajectory recognition device 1100 can be used in conjunction with the above-mentioned Figure 10Corresponding to the agricultural machinery trajectory recognition device 1000, the functions of the processing unit 1010 in the agricultural machinery trajectory recognition device 1000 can be implemented by the processor 1110, the functions of the transceiver unit can be implemented by the transceiver 1130, and the functions of the storage unit 1020 can be implemented by the memory 1120. To avoid repetition, detailed descriptions are omitted here.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0126] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0127] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] In addition, the functional modules in the various embodiments of this application can be integrated into a processing device, or each module can exist physically separately, or two or more modules can be integrated into a device.

[0129] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A deep learning-based agricultural machine trajectory recognition method, characterized in that, The method comprises: acquiring a plurality of trajectory points, each of the plurality of trajectory points comprising position information, speed information, time information and mileage information; determining a first three-channel trajectory graph according to the plurality of trajectory points, the first three-channel trajectory graph comprising a speed graph, a time graph and a mileage graph, the speed graph comprising the speed information of the plurality of trajectory points, the time graph comprising the time information of the plurality of trajectory points, and the mileage graph comprising the mileage information of the plurality of trajectory points; processing the first three-channel trajectory graph using a pre-trained neural network model to determine the type of different trajectory segments of the first three-channel trajectory graph, the type of different trajectory segments of the first three-channel trajectory graph comprising a road segment trajectory and a farmland segment trajectory, or an error trajectory segment.

2. The method of claim 1, wherein, The first three-channel trajectory graph comprises different trajectory segments and a background in the first three-channel trajectory graph, the background being outside the trajectory in the first three-channel trajectory graph, the neural network model comprising a post-processing layer, the post-processing layer determining the background through matrix operation of a first vector and determining the type of different trajectory segments through matrix operation of a second vector, the first vector and the second vector being binary vectors with a length of 3.

3. The method of claim 1, wherein, Before the processing of the first three-channel trajectory graph using the pre-trained neural network model to determine the type of different trajectory segments in the first three-channel trajectory graph, the method further comprises training the neural network model to obtain the pre-trained neural network model. The training of the neural network model comprises: acquiring a second three-channel trajectory graph; training the neural network model using the second three-channel trajectory graph with added labels, wherein the labels correspond to the type of different trajectory segments of the second three-channel trajectory graph.

4. The method of claim 3, wherein, The training of the neural network model further comprises: correcting the neural network model using a loss function, the loss function satisfying, where p t represents the probability that a prediction sample belongs to 1, p t ∈ [0, 1], γ is an adjustment factor, γ > 0, and a t ∈ [2.5, 3].

5. The method of any one of claims 1-4, wherein, The farmland segment trajectory comprises a work area and a turning area, and after the processing of the first three-channel trajectory graph using the pre-trained neural network model to determine the type of different trajectory segments in the first three-channel trajectory graph, the method further comprises determining a work area according to the turning area, the work area being an area enclosed by the turning area.

6. The method of any one of claims 1-4, wherein, The error trajectory segment has a speed difference with an adjacent trajectory segment in time greater than a first threshold value, and / or the error trajectory segment has a mileage difference with an adjacent trajectory segment in time greater than a second threshold value, and the neural network model is further configured to exclude the error trajectory segment from the first three-channel trajectory graph.

7. A device for identifying the trajectory of agricultural machinery, characterized in that, The method comprises: a processing unit and a storage unit, the storage unit is configured to store a neural network model, and the processing unit is configured to: acquire a plurality of trajectory points, each of the plurality of trajectory points comprising position information, speed information, time information and mileage information; determine a first three-channel trajectory graph according to the plurality of trajectory points, the first three-channel trajectory graph comprising a speed graph, a time graph and a mileage graph, the speed graph comprising speed information of the plurality of trajectory points, the time graph comprising time information of the plurality of trajectory points, and the mileage graph comprising mileage information of the plurality of trajectory points; process the first three-channel trajectory graph using the neural network model to determine a type of different trajectory segments of the first three-channel trajectory graph, the type of different trajectory segments of the first three-channel trajectory graph comprising a road segment trajectory and a farmland segment trajectory, or an error trajectory segment.

8. The apparatus of claim 7, wherein, the first three-channel trajectory graph comprises the different trajectory segments and a background of the first three-channel trajectory graph, the background being outside the trajectory in the first three-channel trajectory graph, the neural network model comprises a post-processing layer, the post-processing layer determines the background through matrix operation of a first vector and determines the type of different trajectory segments through matrix operation of a second vector, the first vector and the second vector being binary vectors with a length of 3.

9. The apparatus of claim 7, wherein, Before processing the first three-channel trajectory graph using the pre-trained neural network model to determine the type of different trajectory segments in the first three-channel trajectory graph, the processing unit is further configured to train the neural network model to obtain the pre-trained neural network model. The training of the neural network model comprises: obtain a second three-channel trajectory graph; train the neural network model using the second three-channel trajectory graph with added labels, wherein the labels correspond to the type of different trajectory segments of the second three-channel trajectory graph.

10. The apparatus of claim 9, wherein, The training of the neural network model further comprises: correct the neural network model using a loss function, the loss function satisfying, where p t denotes the probability that a prediction sample belongs to 1, p t ∈ [0, 1], γ is a tuning factor, γ > 0, a t ∈ [2.5, 3].

11. The apparatus of any one of claims 7-10, wherein, the farmland segment trajectory comprises a work area and a turning area, after processing the first three-channel trajectory graph using the pre-trained neural network model to determine the type of different trajectory segments of the first three-channel trajectory graph, the processing unit is further configured to determine a work area according to the turning area, the work area being an area enclosed by the turning area.

12. The apparatus of any one of claims 7-10, wherein, the error trajectory segment has a speed difference with an adjacent trajectory segment in time greater than a first threshold value, and / or the error trajectory segment has a mileage difference with an adjacent trajectory segment in time greater than a second threshold value, the processing unit is further configured to exclude the error trajectory segment from the first three-channel trajectory graph.

13. A computer readable medium characterized by comprise instructions that, when executed on a processor, cause the processor to perform the method of any one of claims 1 to 6. comprise instructions that, when executed on a processor, cause the processor to perform the method of any one of claims 1 to 6.

Images