Information extraction model training method, object information extraction method, device and system
By training a model to recognize parallel strips formed by straight lines using Hough line detection, the method addresses high complexity and poor applicability in identifying linear objects, enhancing recognition efficiency and versatility.
Patent Information
- Application Number
- CN202310668037.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-06
- Publication Date
- 2024-12-06
AI Technical Summary
Existing methods for identifying linear or strip-shaped objects, such as cotton rows or obstacles like wires and poles, suffer from high computational complexity and poor applicability due to the neural network models learning the complex characteristics of these objects directly.
An iterative training method using Hough line detection to create learning samples, focusing on the characteristics of parallel strips formed by straight lines, reduces complexity by training a model to recognize the position of these strips without needing to identify the object type, thus requiring fewer samples and simplifying the network structure.
This approach significantly reduces computational complexity and improves applicability by enabling the model to recognize any linear or strip-shaped object, making it suitable for various scenarios like cotton field management and drone navigation.
Smart Images

Figure CN119091242A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an information extraction model training method, an object information extraction method, a device and a system. Background Art
[0002] In some intelligent systems, linear or strip-shaped objects need to be identified. For example, in the intelligent management scenario of cotton fields, cotton rows need to be identified so that the identified cotton rows can be used for seedling rate statistics, weed identification, operation navigation, etc. For another example, in the flight control scenario of drones, linear or strip-shaped obstacles such as wires and poles need to be identified so that obstacle avoidance can be performed based on the identified wires and poles.
[0003] In the prior art, a neural network model can be trained, and an image of the area to be identified is input into the neural network model, and the neural network model identifies and outputs the position of the linear or strip-shaped object in the image.
[0004] However, the existing methods have the problems of high computational complexity and poor applicability. Summary of the invention
[0005] The purpose of this application is to provide an information extraction model training method, object information extraction method, device and system to address the deficiencies in the above-mentioned prior art, so as to solve the problems of high computational complexity and poor applicability in the prior art.
[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0007] In a first aspect, the present application provides an information extraction model training method, comprising:
[0008] The initial extraction model is iteratively trained using a training sample set until a preset iteration stop condition is met; wherein the training sample set includes a learning sample represented by a straight line detection result and label information obtained by annotating the straight line detection result; the learning sample is obtained by performing Hough line detection on an original image, and the original image includes a linear or strip-shaped target object; the label information is used to represent the real position of the target object in the original image;
[0009] When the initial extraction model meets a preset iteration stop condition, the current initial extraction model is determined as the information extraction model.
[0010] As a possible implementation method, the iterative training of the initial extraction model using the training sample set includes:
[0011] Inputting a learning sample into the current initial extraction model to obtain a current prediction result output by the current initial extraction model based on the learning sample;
[0012] The current loss function is constructed based on the current prediction result, the current label information and the preset sample weights;
[0013] The current loss function is used to back-propagate the current initial extraction model to update the model parameters of the current initial extraction model.
[0014] As a possible implementation method, the current loss function is constructed according to the current prediction result, the current label information and the preset sample weight, including:
[0015] Determine the intersection-over-union ratio of the current label information and the current prediction result;
[0016] Using the sample weight as an influencing factor, determining the cross entropy between the current label information and the current prediction result;
[0017] Based on the intersection-over-union ratio and the cross entropy, a current loss function is constructed.
[0018] As a possible implementation, the training sample set is obtained by the following steps:
[0019] Get multiple original images;
[0020] Performing image transformation on the multiple original images respectively to obtain multiple transformed images, where the transformed images are binary images or mask images of the original images;
[0021] Performing Hough line detection on the multiple transformed images respectively to obtain multiple line detection results, and using the multiple line detection results as the learning samples; wherein the line detection results include multiple rows of elements, each element corresponds to a straight line in the original image, the position of each element is used to represent the position of the corresponding straight line in the original image, and the element value of each element is used to represent the reliability of the straight line;
[0022] According to the element values of the multiple rows of elements, key result information for characterizing the real position of the target object in the original image is determined, and the key result information is used as the label information.
[0023] As a possible implementation manner, determining the key result information for characterizing the real position of the target object in the original image according to the element values of the multiple rows of elements includes:
[0024] Determine multiple key factor information according to the element values of the multiple rows of elements, the key factor information including: gap density and strip element average value;
[0025] The key result information is determined according to the plurality of key factor information.
[0026] As a possible implementation, the multiple rows of elements are stored in a two-dimensional array, the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines passing through the point corresponding to the position in the two-dimensional array in the Hough space;
[0027] The determining of multiple key factor information according to the element values of the multiple rows of elements includes:
[0028] Determine the stripe elements and the non-stripe elements in each row according to the element values of the elements in each row of the two-dimensional array;
[0029] Calculate the ratio of the number of non-strip elements in each row to the sum of the number of strip elements and non-strip elements in each row to obtain the void density of each row;
[0030] The average values of the element values of the strip elements in each row are calculated respectively to obtain the average value of the strip elements in each row.
[0031] As a possible implementation manner, determining the key result information according to the multiple key factor information includes:
[0032] Calculate the product of the void density of each row and the average value of the strip elements to obtain the representation degree value of each row;
[0033] Each element of the row with the largest representation degree value is used as the key result information.
[0034] As a possible implementation manner, after determining the key result information, the method further includes:
[0035] Obtain at least one continuous element sequence according to the element value of each element in the row where the key result information is located;
[0036] Selecting a threshold value according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold value of the continuous element sequence;
[0037] Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filtering processing is performed on the elements in the continuous element sequence to obtain denoised key result information.
[0038] As a possible implementation manner, the Hough line detection is performed on the multiple transformed images respectively to obtain multiple line detection results, including:
[0039] Taking the center point of each transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the line detection result.
[0040] As a possible implementation manner, in the initial extraction model, there is at least one hole convolution layer; and / or, in the initial extraction model, the initialization convolution parameter of the last layer is a preset value.
[0041] As a possible implementation manner, the convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
[0042] As a possible implementation, the original image includes any one of the following: an image containing crop rows, an image of electric poles, an image of electric wires;
[0043] The recognition result obtained by the information extraction model processing the original image containing the target object includes: the position information of the target object in the original image.
[0044] In a second aspect, the present application provides an object information extraction method, comprising:
[0045] Performing image transformation on the image to be identified to obtain a transformed image, wherein the transformed image is a binary image or a mask image of the image to be identified;
[0046] Performing Hough line detection on the transformed image to obtain a line detection result;
[0047] The straight line detection result is input into a pre-trained information extraction model to obtain target object information in the image to be identified; the target object information includes: position information of the target object in the image to be identified, wherein the information extraction model is trained based on the method described in the first aspect above.
[0048] In a third aspect, the present application provides an information extraction model training device, comprising:
[0049] A training module, used for iteratively training the initial extraction model using a training sample set until a preset iteration stop condition is met; wherein the training sample set includes a learning sample represented by a straight line detection result, and label information obtained by annotating based on the straight line detection result; the learning sample is obtained by performing Hough line detection on an original image, and the original image includes a linear or strip-shaped target object; the label information is used to represent the real position of the target object in the original image;
[0050] The determination module is used to determine the current initial extraction model as the information extraction model when the initial extraction model meets a preset iteration stop condition.
[0051] As a possible implementation, the training module is specifically used for:
[0052] Inputting a learning sample into the current initial extraction model to obtain a current prediction result output by the current initial extraction model based on the learning sample;
[0053] The current loss function is constructed based on the current prediction result, the current label information and the preset sample weights;
[0054] The current loss function is used to back-propagate the current initial extraction model to update the model parameters of the current initial extraction model.
[0055] As a possible implementation, the training module is specifically used for:
[0056] Determine the intersection-over-union ratio of the current label information and the current prediction result;
[0057] Using the sample weight as an influencing factor, determining the cross entropy between the current label information and the current prediction result;
[0058] Based on the intersection-over-union ratio and the cross entropy, a current loss function is constructed.
[0059] As a possible implementation, the training module is further used for:
[0060] Get multiple original images;
[0061] Performing image transformation on the multiple original images respectively to obtain multiple transformed images, where the transformed images are binary images or mask images of the original images;
[0062] Performing Hough line detection on the multiple transformed images respectively to obtain multiple line detection results, and using the multiple line detection results as the learning samples; wherein the line detection results include multiple rows of elements, each element corresponds to a straight line in the original image, the position of each element is used to represent the position of the corresponding straight line in the original image, and the element value of each element is used to represent the reliability of the straight line;
[0063] According to the element values of the multiple rows of elements, key result information for characterizing the real position of the target object in the original image is determined, and the key result information is used as the label information.
[0064] As a possible implementation, the training module is specifically used for:
[0065] Determine multiple key factor information according to the element values of the multiple rows of elements, the key factor information including: gap density and strip element average value;
[0066] The key result information is determined according to the plurality of key factor information.
[0067] As a possible implementation, the multiple rows of elements are stored in a two-dimensional array, the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines passing through the point corresponding to the position in the two-dimensional array in the Hough space;
[0068] The training module is specifically used for:
[0069] Determine the stripe elements and the non-stripe elements in each row according to the element values of the elements in each row of the two-dimensional array;
[0070] Calculate the ratio of the number of non-strip elements in each row to the sum of the number of strip elements and non-strip elements in each row to obtain the void density of each row;
[0071] The average values of the element values of the strip elements in each row are calculated respectively to obtain the average value of the strip elements in each row.
[0072] As a possible implementation, the training module is specifically used for:
[0073] Calculate the product of the void density of each row and the average value of the strip elements to obtain the representation degree value of each row;
[0074] Each element of the row with the largest representation degree value is used as the key result information.
[0075] As a possible implementation method, the training module is specifically used to:
[0076] Obtain at least one continuous element sequence according to the element value of each element in the row where the key result information is located;
[0077] Selecting a threshold value according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold value of the continuous element sequence;
[0078] Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filtering processing is performed on the elements in the continuous element sequence to obtain denoised key result information.
[0079] As a possible implementation, the training module is specifically used for:
[0080] Taking the center point of each transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the line detection result.
[0081] As a possible implementation manner, in the initial extraction model, there is at least one hole convolution layer; and / or, in the initial extraction model, the initialization convolution parameter of the last layer is a preset value.
[0082] As a possible implementation manner, the convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
[0083] As a possible implementation, the original image includes any one of the following: an image containing crop rows, an image of electric poles, an image of electric wires;
[0084] The recognition result obtained by the information extraction model processing the original image containing the target object includes: the position information of the target object in the original image.
[0085] In a fourth aspect, the present application provides an object information extraction device, comprising:
[0086] A transformation module, used for performing image transformation on the image to be identified to obtain a transformed image, wherein the transformed image is a binary image or a mask image of the image to be identified;
[0087] A detection module, used for performing Hough line detection on the transformed image to obtain a line detection result;
[0088] An extraction module is used to input the straight line detection result into a pre-trained information extraction model to obtain the target object information in the image to be identified; the target object information includes: the position information of the target object in the image to be identified, wherein the information extraction model is trained based on the method described in the first aspect above.
[0089] In a fifth aspect, the present application provides a smart agricultural system, including:
[0090] An acquisition device, used for acquiring an original image;
[0091] A processing device, used to process the original image obtained by the acquisition device according to the object information extraction method according to the second aspect of claim 1, so as to obtain the target object information in the original image;
[0092] A graphical interface device is used to generate display information of the corresponding original object according to the target object information.
[0093] In a sixth aspect, the present application provides an electronic device comprising: a processor and a memory, the memory storing machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to execute the steps of the information extraction model training method described in the first aspect above or the steps of the object information extraction method described in the second aspect above.
[0094] In the seventh aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the information extraction model training method described in the first aspect or the steps of the object information extraction method described in the second aspect are executed.
[0095] The information extraction model training method, object information extraction method, device and system provided by the present application, in the iterative training process of the initial extraction model, the straight line detection result of the original image is used as a training sample, and the information label information that can characterize the real position of the target object is obtained by annotating the straight line detection result, so that the initial extraction model can learn to obtain the recognition ability of the position of the parallel strips composed of the straight lines. Compared with the learning of the characteristics of the linear or ribbon-shaped object itself in the prior art, this embodiment focuses on the learning of the characteristics of the parallel strips formed by the straight lines. Obviously, the complexity of the characteristics of the parallel strips is lower, so the network structure and network parameters of the initial extraction model can be made less complex, and at the same time, the training samples required to be used are also less, so the complexity of the operation can be greatly reduced. In addition, the initial extraction model only needs to learn the recognition ability of the position of the parallel strips composed of the straight lines, without paying attention to the type of objects of the parallel strips, so it can be applied to the recognition of any linear or ribbon-shaped object, so the applicability of the method can be greatly improved.
[0096] In addition, when performing information extraction, after the straight line detection results are input into the information extraction model, the position information of the target object in the image to be identified can be obtained. Since the information extraction model has the ability to identify parallel strips, the complexity of the features of parallel strips is lower than that of the features of the object itself. Therefore, it can reduce the complexity of the calculation and greatly improve the applicability. BRIEF DESCRIPTION OF THE DRAWINGS
[0097] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0098] Figure 1A flowchart of the information extraction model training method provided in the embodiment of the present application;
[0099] Figure 2 A schematic diagram of an iterative training process of an information extraction model training method provided in an embodiment of the present application;
[0100] Figure 3 A schematic diagram of a process for constructing a current loss function for the information extraction model training method provided in an embodiment of the present application;
[0101] Figure 4 A schematic diagram of a process for obtaining a training sample set in the information extraction model training method provided in an embodiment of the present application;
[0102] Figure 5 is a schematic diagram of a transformed image obtained after the original image is binarized;
[0103] Figure 6 is a visualization diagram of a straight line detection result obtained through the Hough straight line detection of this embodiment;
[0104] Figure 7 A visualization diagram of the line detection result obtained when the origin of the polar coordinate system is set at a vertex of the image;
[0105] Figure 8 A schematic diagram of the process to determine key outcome information;
[0106] Fig. 9 Schematic diagram of the process for determining void density and average values of strip elements;
[0107] Fig.10 A schematic diagram of the process of filtering key result information;
[0108] Fig.11 A flowchart of an object information extraction method provided in an embodiment of the present application;
[0109] Fig.12 It is a visualization diagram of the location information obtained based on the information extraction model;
[0110] Fig.13 A module structure diagram of the information extraction model training device provided in an embodiment of the present application;
[0111] Fig.14 A module structure diagram of an object information extraction device provided in an embodiment of the present application;
[0112] Fig.15 A schematic diagram of the structure of an electronic device 150 provided in an embodiment of the present application. DETAILED DESCRIPTION
[0113] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0114] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0115] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0116] In the prior art, linear or ribbon-shaped objects are generally identified by training a neural network model. This method is to construct a large number of learning samples and mark the linear or ribbon-shaped objects in the learning samples. For example, for the identification of cotton rows in a cotton field, the prior art needs to mark the edges of the cotton rows in the image. Furthermore, in the process of training the neural network model, the neural network model can learn the characteristics of the linear or ribbon-shaped objects in the learning samples based on the learning samples, so that after the neural network model training is completed, it has the ability to identify linear or ribbon-shaped objects in the image.
[0117] However, the essence of the solution of the prior art is that the neural network model constructs the ability to identify the characteristics of the linear or ribbon-shaped object itself. The above-mentioned solution of the prior art essentially leads to the problem that the neural network has high complexity and poor applicability. Specifically, first of all, the characteristics of the linear or ribbon-shaped object itself are highly complex, so the neural network model needs to learn and process the characteristics of the linear or ribbon-shaped object itself through a complex network structure and network parameters. In addition, in order to improve and ensure the recognition ability of the neural network model, a large number of learning samples need to be labeled to train the neural network model. As a result, the complexity of the neural network model in terms of training process, network structure, etc. is too high. Secondly, the neural network model can only learn the learning ability of the field where the learning sample is located, and cannot apply this ability to other fields. For example, if a labeled cotton field image is used as a training sample, the trained neural network model can identify the cotton rows in the cotton field, but cannot identify the telephone poles. Therefore, the applicability of the existing neural network model is poor.
[0118] Based on the above problems, the present application proposes an object information extraction method. This method abandons the existing idea of linear or ribbon-shaped object recognition based on the neural network model learning the characteristics of the linear or ribbon-shaped objects themselves, and instead is based on the commonality that linear or ribbon-shaped objects can form straight lines. By training the neural network model to recognize linear or ribbon-shaped objects, accurate recognition of linear or ribbon-shaped objects can be achieved, thereby significantly reducing the complexity of recognition and significantly improving the applicability of recognition.
[0119] The embodiments of the present application can be applied to any scenario where linear or strip-shaped objects need to be identified. For example, in the intelligent management scenario of cotton fields, the method of the embodiments of the present application can be used to identify the planting area of the cotton row, so that the identified planting area of the cotton row can be used to perform emergence rate statistics, weed identification, operation navigation, etc. For another example, in the flight control scenario of a drone, the method of the embodiments of the present application can be used to identify linear or strip-shaped obstacles such as wires and poles, so that obstacle avoidance can be performed based on the identified wires, poles, etc.
[0120] Figure 1 This is a flow chart of the information extraction model training method provided in the embodiment of the present application. The execution subject of the method can be any electronic device with computing and processing capabilities, such as a desktop computer, a server, etc. Figure 1 As shown, the method includes:
[0121] S101. Iteratively train the initial extraction model using a training sample set until a preset iteration stop condition is met; wherein the training sample set includes learning samples represented by straight line detection results, and label information obtained by annotation based on the straight line detection results; the learning samples are obtained by performing Hough line detection on the original image, and the original image includes a linear or strip-shaped target object; the label information is used to represent the actual position of the target object in the original image.
[0122] Optionally, the above-mentioned initial extraction model can be, for example, a convolutional neural network model with semantic segmentation function. Exemplarily, the initial extraction model can be a fully convolutional neural network model (Fully Convolutional Network, referred to as FCN) or a U-Net network model. The embodiment of the present application does not limit the specific form of the initial extraction model.
[0123] Optionally, the above training sample set can be constructed and the initial extraction model can be iteratively trained using the training sample set. It is worth noting that during the iterative training process, only the pre-constructed training sample set can be used, or the training sample set can be adjusted or expanded according to actual needs to continuously optimize the learning ability of the initial extraction model.
[0124] Optionally, the training sample set may include multiple learning samples as described above, and for each learning sample, the label information of the learning sample is also given in the training sample set. The learning sample and its label information are bound as a group of samples and stored in the training sample set. That is, each group of samples includes a learning sample and the label information of the learning sample.
[0125] In each group of samples, the learning samples are obtained by performing Hough line detection on the original image. Specifically, the original image includes a linear or band-shaped target object. The target object may be, for example, a cotton row, an electric wire, an electric pole, etc., and correspondingly, the original object may be an image containing a cotton row, an electric wire, or an electric pole. After acquiring the original image, a Hough line detection is performed on the original image to obtain a line detection result. As an example, the line detection result may be presented in the form of a two-dimensional array, and the line information in the original image may be obtained through the element position and element value in the two-dimensional array. After obtaining the line detection result, the line detection result may be used as a learning sample in the above group of samples.
[0126] At the same time, by marking the straight line detection results, the label information of the learning sample can be obtained. The label information can characterize the real position of the target object in the original image. As an example, the label information can include some elements in the above-mentioned two-dimensional array, and the real position of the target object in the original image can be easily known by using these elements. Optionally, when marking the straight line detection results, it can be marked manually, or it can also be marked automatically. Among them, the specific process of automatic marking will be described in detail in the following embodiments.
[0127] S102: When the initial extraction model satisfies a preset iteration stop condition, the current initial extraction model is determined as the information extraction model.
[0128] Optionally, in each iteration of the iterative training of the initial extraction model, the learning samples in each group of samples are input into the initial extraction model, and the actual results are predicted by the initial extraction model. By comparing the actual results with the label information of the learning samples, it is determined whether the iteration stopping condition is met. When the iteration stopping condition is met, the initial extraction model at the last iteration is used as the above-mentioned information extraction model.
[0129] After obtaining the above information extraction model, the information extraction model can be used to extract the parallel stripe information in the image, and the position of the parallel stripe in the image can be obtained accordingly. This extraction process will be described in detail in the following embodiments.
[0130] It is worth noting that the input information of the initial extraction model in the embodiment of the present application is not the original image, but the line detection result obtained by performing Hough line detection on the original image. Correspondingly, the output result of the initial extraction model is the information representing the real position of the linear or band-shaped object in the original image.
[0131] In this embodiment, during the iterative training process of the initial extraction model, the straight line detection results of the original image are used as training samples, and the information label information that can characterize the real position of the target object is obtained by annotating the straight line detection results, so that the initial extraction model can learn to obtain the ability to recognize the position of the parallel strips composed of straight lines. Compared with the learning of the characteristics of the linear or ribbon-shaped objects themselves in the prior art, this embodiment focuses on the learning of the characteristics of the parallel strips formed by the straight lines. Obviously, the complexity of the characteristics of the parallel strips is lower, so the complexity of the network structure and network parameters of the initial extraction model can be lower. At the same time, the training samples required are also less, so the complexity of the operation can be greatly reduced. In addition, the initial extraction model only needs to learn the ability to recognize the position of the parallel strips composed of straight lines, without paying attention to the type of objects of the parallel strips. Therefore, it can be applied to the recognition of any linear or ribbon-shaped objects, so the applicability of the method can be greatly improved.
[0132] As mentioned above, the embodiments of the present application can be applied to any scene where linear or strip-shaped objects need to be identified. Therefore, the objects targeted by the embodiments of the present application can be any linear or strip-shaped objects. Exemplarily, the objects can be crop rows, electric poles, electric wires, etc. Correspondingly, the original image described in the embodiments of the present application can be an image containing crop rows, an electric pole image, or an electric wire image, etc.
[0133] Correspondingly, the recognition result obtained by the information extraction model processing the original image containing the target object may include the position information of the target object in the original image.
[0134] Specifically, in an optional manner, the above-mentioned position information may be key result information characterizing the position of the target object in the original image. The key result information may be the part of the straight line detection result of the input information extraction model that best characterizes the parallel strips. The position of the target object in the original image may be calculated using the key result information. In another optional manner, the above-mentioned position information may include the angle and width of the parallel strips in the original image. Based on the angle and width, the position of the parallel strips in the original image may be known.
[0135] Optionally, before iteratively training the above initial extraction model, the initial extraction model may be first built and initialized.
[0136] As an optional implementation, the constructed initial extraction model may include at least one hole convolution layer; and / or, in the initial extraction model, the initialization convolution parameters of the last layer are preset values.
[0137] When initializing the initial extraction model, the initialization convolution parameters of the last layer can be specified. In the present application, by specifying the initialization convolution parameter value of the last layer as a preset value rather than a random value, the convergence speed of the initial extraction model can be faster.
[0138] As an example, the convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
[0139] By specifying the initial value of the bias as a negative number and the initial value of the weight as 0, the gradient explosion caused by too few positive samples in the training sample set can be avoided, making the initial extraction model converge faster.
[0140] The following is an explanation of the process of iteratively training the initial extraction model based on the training sample set. Figure 2 The processing flow shown is an iterative process in the iterative training of the initial extraction model. Before the initial extraction model meets the iteration stop condition, it can go through multiple iterations. Each iteration can be performed as follows Figure 2 Process shown.
[0141] Figure 2 A schematic diagram of an iterative training process of the information extraction model training method provided in an embodiment of the present application, such as Figure 2 As shown, an optional manner of the above step S101 includes:
[0142] S201. Input a learning sample into the current initial extraction model to obtain a current prediction result output by the current initial extraction model based on the learning sample.
[0143] The current initial extraction model refers to an initial extraction model that has not been iterated, or an initial extraction model that has been updated in the last iteration. The current initial extraction model is used as the initial extraction model in this iteration.
[0144] Optionally, all or part of the learning samples in the training sample set can be selected, and the selected learning samples can be input into the current initial extraction model one by one, and the current initial extraction model can output the current prediction result accordingly. Among them, the label information of the learning samples selected in this iteration belongs to the current label information of this iteration. Consistent with the recognition result output by the above-mentioned information extraction model, the current prediction result output by the current initial extraction model may include the position information of the target object in the original image, and the position information may be the aforementioned key result information, or the angle and width of the parallel strips.
[0145] It should be understood that the current prediction result is consistent with the form of the label information of the learning sample. Specifically, if the current prediction result is key result information, the label information can also be key result information. If the current prediction result is the angle and width of the parallel stripes, the label information can also be the angle and width of the parallel stripes.
[0146] S202: construct a current loss function based on the current prediction result, the current label information and the preset sample weights.
[0147] The sample weights mentioned above may refer to the weights of positive samples or negative samples in each learning sample. Positive samples refer to the parts of the learning sample that belong to parallel strips, and negative samples refer to the parts of the learning sample that do not belong to parallel strips. For linear or strip-shaped object recognition scenarios, the number of negative samples is generally much larger than the number of positive samples. Therefore, the preset sample weights mentioned above can make the weights of positive samples higher, thereby avoiding the problem of inaccurate current loss function caused by imbalance of positive and negative samples.
[0148] Optionally, the above preset sample weights can be used as constraints to perform comparative analysis on the current prediction results and the current label information, thereby constructing the current loss function.
[0149] S203: Back-propagate the current initial extraction model using the current loss function to update the model parameters of the current initial extraction model.
[0150] Optionally, the current loss function can be used to determine whether the current initial extraction model meets the above-mentioned iteration stop condition. If so, the aforementioned step S102 can be entered to obtain the information extraction model. Otherwise, the model parameters of the current initial extraction model can be updated according to the current loss function, and the updated current initial extraction model can be used for the next iteration.
[0151] In this embodiment, based on the current prediction result output by the current initial extraction model, the current label information and the preset sample weights, the current loss function can be constructed, and the current initial extraction model is back-propagated accordingly. Since the preset sample weights are used, the weight of the positive sample is higher, thereby avoiding the problem of inaccurate current loss function caused by the imbalance of positive and negative samples.
[0152] Figure 3 A schematic diagram of a process for constructing a current loss function for the information extraction model training method provided in an embodiment of the present application, such as Figure 3 As shown, an optional manner of the above step S202 includes:
[0153] S301: Determine the intersection-over-union ratio of current label information and current prediction result.
[0154] Optionally, taking the current label information and the current prediction result as the aforementioned key result information as an example, the key result information can be presented in the form of a matrix, and the intersection and union of the matrix of the current label information and the matrix of the current prediction result can be calculated respectively, and the ratio of the intersection and the union can be calculated to obtain the above-mentioned intersection-union ratio.
[0155] Exemplarily, the intersection over union (IOU) of the current label information and the current prediction result may be calculated based on an intersection over union (IOU) function.
[0156] S302: Using the sample weight as an influencing factor, determine the cross entropy between the current label information and the current prediction result.
[0157] Optionally, continuing with the example where the current label information and the current prediction result are the aforementioned key result information, and the key result information is presented in the form of a matrix, the cross entropy between the matrix of the current label information and the matrix of the current prediction result can be calculated using the above sample weights as coefficients.
[0158] S303: Based on the above intersection-over-union ratio and the above cross entropy, a current loss function is constructed.
[0159] As an example, the sum of the above-mentioned intersection-over-union ratio and cross entropy can be calculated, and the sum of the intersection-over-union ratio and the cross entropy can be used as the result of the current loss function.
[0160] In this embodiment, by determining the intersection-over-union ratio and cross entropy of the current label information and the current prediction result, the extraction of positive samples in the learning samples can be guaranteed, so that the model can better adapt to the unbalanced sample distribution, thereby greatly improving the convergence speed, performance and stability of the initial extraction model.
[0161] The process of automatically acquiring a training sample set is described below. It should be understood that automatically acquiring a training sample set includes automatically acquiring a learning sample in each group of samples and automatically annotating a straight line detection result to obtain label information.
[0162] Figure 4 A flowchart of the information extraction model training method for obtaining a training sample set provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the aforementioned training sample set can be obtained through the following steps:
[0163] S401, obtaining multiple original images.
[0164] S402 , performing image transformation on the plurality of original images respectively to obtain a plurality of transformed images, where the transformed images are binary images or mask images of the original images.
[0165] In an optional manner, the original image can be subjected to a binary transformation, and accordingly, the transformed image obtained is the above-mentioned binary image. Specifically, in the process of binary transformation, the grayscale value of each pixel point can be transformed to 0 or 255 according to the original pixel value of each pixel point on the original image, so that the transformed image presents only black and white effects, which is convenient for subsequent straight line detection. In another optional manner, the original image can be subjected to mask processing, and accordingly, the transformed image obtained is the above-mentioned mask image. Specifically, before the mask processing, an image mask can be generated by specifying data values, data ranges, regions of interest, etc. The image mask can be a two-dimensional matrix array or a multi-valued image, etc. Furthermore, during the mask processing, the image mask is multiplied by the original image to obtain a transformed image, so that part of the region in the transformed image is shielded, and only the region of interest is retained for subsequent straight line detection.
[0166] S403, performing Hough line detection on the plurality of transformed images to obtain a plurality of line detection results, and using the plurality of line detection results as the learning samples. The line detection results include a plurality of rows of elements, each element corresponding to a line in the original image, the position of each element is used to represent the position of the corresponding line in the original image, and the element value of each element is used to represent the reliability of the line.
[0167] Optionally, the coordinate system where the transformed image is located has the following mapping relationship with the Hough coordinates of the Hough space: each point in the original coordinate system can be mapped to a straight line in the Hough coordinate system. Similarly, each point in the Hough coordinate system can also be mapped to a straight line in the original coordinate system. Accordingly, a straight line in the original coordinate system can be evaluated by the number of lines that intersect at a point in the Hough coordinate system. Specifically, in the Hough coordinate system, the more straight lines passing through a point, it indicates that the line mapped to the point in the original coordinate system is composed of more points, and the greater the probability that the line mapped to the point in the original coordinate system is a straight line.
[0168] Optionally, the original coordinate system of the original image may be a polar coordinate system or a Cartesian coordinate system.
[0169] Based on the above mapping relationship, when performing Hough line detection on the transformed image, all points in the original coordinate system can be first mapped to a straight line in the Hough coordinate system using the mapping relationship, and then the points where the straight lines intersect can be judged. If the number of straight lines passing through a certain point reaches a threshold, the point can be mapped to a line in the original coordinate system as a straight line in the original image. After the Hough line detection is completed, the above line detection result can be obtained, and the above line detection result can be used as a learning sample. Among them, the above line detection result includes the above multiple rows of elements. Exemplarily, the line detection result can be presented in the form of a two-dimensional array, which includes multiple rows of elements, and the number of elements in each row of the two-dimensional array can be one or more. For each element of each row in the two-dimensional array, its position in the matrix can represent the position of a straight line corresponding to the element in the original image. Among them, the position of the element in the matrix can refer to the row where the element is located and the column where the element is located. At the same time, the element value of the element can represent the reliability of the straight line corresponding to the element. Exemplarily, the larger the element value of the element, the higher the reliability of the straight line corresponding to the element. It should be understood that the higher the reliability, the greater the possibility of belonging to the straight line, and the more reliable the result.
[0170] S404: Determine key result information for characterizing the real position of the target object in the original image according to the element values of the multiple rows of elements, and use the key result information as the label information.
[0171] Since each element in the above-mentioned multiple lines of elements corresponds to a straight line in the original image, and the element value of each element can characterize the reliability of the straight line, the above-mentioned key result information can be obtained by analyzing and comparing the element values of the multiple lines of elements based on the element values of the multiple lines of elements. Optionally, the above-mentioned key result information can include some elements determined from the above-mentioned multiple lines of elements, and these elements are sufficient to identify parallel strips in the original image. Therefore, the above-mentioned key result information can be used as label information.
[0172] As another possible implementation, after obtaining the above key result information, the angle and width of the parallel strips can be extracted based on the key result information and the position of at least one row of elements in the multiple rows of elements, and the angle and width can be used as label information.
[0173] As an optional implementation, when performing Hough line detection on the transformed image in the above step S403, the process may be as follows.
[0174] Taking the center point of the transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the above line detection result.
[0175] Optionally, when performing Hough line detection on the transformed image, if the origin of the polar coordinate system is set at other positions of the image, such as the left vertex of the image, the line parameters in the line detection results will all fall into the same quadrant, resulting in interlaced and mixed line detection results, which will make it difficult to extract subsequent key result information. Considering this problem, in this embodiment, the center point of the transformed image is used as the origin of the polar coordinate system, so that the line parameters can be evenly distributed, and the line parameters corresponding to the lines at the same angle can be concentrated in a certain area, so that the key result information can be obtained more conveniently.
[0176] Figure 5 is a schematic diagram of the transformed image obtained after the original image is binarized, as shown in Figure 5 As shown, each white strip is a parallel strip of an object, and the parallel strip of the object can be, for example, a cotton row, an electric wire, an electric pole, etc. Figure 6 is a visualization diagram of the straight line detection result obtained by the Hough straight line detection of this embodiment, as shown in Figure 6 As shown, the white highlighted portion is the location of the key result information, and the key result information is sufficient to identify the parallel strips in the original image. Figure 7 This is a visualization diagram of the line detection results obtained when the origin of the polar coordinate system is set at a vertex of the image. Figure 7 It can be seen that this method makes the white highlight part tilted, which makes it difficult to extract the subsequent key result information. Figure 6 and Figure 7 ,Obviously, the straight line detection result obtained by taking the center point of the ,transformed image as the origin of the polar coordinate system is more ,conducive to the subsequent key result information extraction.
[0177] The process of determining the key result information in the above step S404 is described below.
[0178] Figure 8 A flow chart to determine key outcome information, such as Figure 8 As shown, an optional manner of the above step S404 includes:
[0179] S801. Determine multiple key factor information according to the element values of the multiple rows of elements, where the key factor information includes: gap density and strip element average value.
[0180] S802: Determine the key result information according to the multiple key factor information.
[0181] As mentioned above, the key result information is sufficient to identify parallel stripes in the original image, and the key result information is the partial elements in the multiple rows of elements that best represent the parallel stripes. In this embodiment, the key result information is found using the key factor information.
[0182] Optionally, the key factor information includes the above-mentioned gap density and strip element average value. Among them, the gap density can refer to the proportion of non-strip elements in multiple elements, and the non-strip elements can refer to elements with element values of zero or less than a threshold. These elements cannot correspond to a real straight line in the original coordinate system due to their small element values. Corresponding to the non-strip elements are strip elements, and strip elements can refer to elements with element values greater than or equal to the threshold. For multiple elements in the straight line detection result, the gap density can be used to measure the interval between parallel strips. Based on this interval, some elements that can best characterize parallel strips can be identified without being affected by some interfering non-strip elements between parallel strips. In addition to the gap density, the key factor information also includes the strip element average value. The strip element average value is obtained by statistics of the element values of the strip elements. The strip element average value can be used to measure the possibility of elements forming parallel strips, and then this possibility can also be used to identify some elements that can best characterize parallel strips.
[0183] Based on the above void density and strip element average values, multiple elements can be analyzed and measured to determine key result information.
[0184] In this embodiment, multiple key factor information can be determined based on the element values of multiple rows of elements. The key factor information includes the gap density and the strip element average value. The gap density can be used to measure the interval between parallel strips, and the strip element average value can be used to measure the possibility of elements forming parallel strips. Therefore, based on the gap density and the strip element average value, key result information with high accuracy can be determined, and the parallel strip information extracted based on the key result information has high accuracy.
[0185] As a possible implementation method, the above-mentioned multiple rows of elements can be stored in a two-dimensional array, and the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines in the Hough space passing through the point corresponding to the position in the two-dimensional array.
[0186] Specifically, each element in the two-dimensional array can correspond to a point in the Hough space, and of course, it also corresponds to a straight line in the original image. Therefore, the position of the element in the two-dimensional array can represent the position of the corresponding straight line in the original image. At the same time, each element has an element value in the two-dimensional array, and the element value can indicate the number of straight lines passing through the point corresponding to the element in the Hough space. The following is an example. Assuming that the coordinate system of the original image is a polar coordinate system, there is an element in the two-dimensional array, and the element is located in the a row and b column of the two-dimensional array, which indicates that the element corresponds to the point (a, b) in the Hough space, and corresponds to the straight line with a polar angle of a and a polar radius of b in the polar coordinate system. Assuming that the element value of this element in the a row and b column is c, it indicates that there are c straight lines passing through the point (a, b) in the Hough space.
[0187] Based on the above storage method of multiple rows of elements, an optional method of the above step S801 includes:
[0188] According to the element values of the elements in each row of the two-dimensional array, the gap density of each row and the average value of the strip elements are determined.
[0189] Optionally, when multiple rows of elements are stored in a two-dimensional array, the gap density of each row and the average value of the strip elements can be determined on a row-by-row basis.
[0190] Taking rows as units and determining the gap density and strip element average value of each row can make the determined gap density and strip element average value highly matched with the parallel strips, thereby making the subsequently determined key result information and parallel strip information highly accurate.
[0191] The following describes the process of determining the void density of each row and the average value of the stripe elements.
[0192] Fig. 9 Schematic diagram of the process for determining the void density and the average value of the strip elements, such as Fig. 9 As shown, an optional method for determining the gap density of each row and the average value of the strip elements according to the element values of the elements in each row includes:
[0193] S901. Determine stripe elements and non-stripe elements in each row according to element values of elements in each row of a two-dimensional array.
[0194] Optionally, as mentioned above, non-strip elements may refer to elements whose element values are zero or less than a threshold value. These elements cannot correspond to a real straight line in the original coordinate system due to their small element values. Correspondingly, strip elements may refer to elements whose element values are greater than or equal to the threshold value. Strip elements may correspond to a real straight line in the original coordinate system. Based on this, in this step, the elements in each row of the two-dimensional array may be judged one by one. If the element value of the element is greater than or equal to the threshold value, the element is treated as a strip element. If the element value of the element is 0 or less than the threshold value, the element is treated as a non-strip element.
[0195] It is worth noting that in the original image, there are generally no objects at the left and right edges of the image. At the same time, there are no objects between the two parallel strips of objects or there are only sporadic interfering objects. Therefore, by determining the strip elements and non-strip elements through this step, the left and right edges of the original image and the middle part that does not belong to the object can be identified.
[0196] S902 , respectively calculating the ratio of the number of non-strip elements in each row to the sum of the number of strip elements and non-strip elements in each row to obtain the gap density of each row.
[0197] As mentioned above, the gap density can refer to the proportion of non-strip elements in multiple elements. Specifically in this step, since it is based on each row, the proportion of the number of non-strip elements in each row to all elements can be calculated to obtain the gap density of each row. Among them, the larger the proportion of the number of non-strip elements, the greater the gap density. Conversely, the smaller the proportion of the number of non-strip elements, the smaller the gap density.
[0198] S903 , respectively calculate the average values of the element values of the strip elements in each row to obtain the average value of the strip elements in each row.
[0199] Exemplarily, the element values of all stripe elements in each row may be added together, and then the added result may be divided by the number of stripe elements to obtain the average value of the stripe elements in each row.
[0200] In this embodiment, the gap density and the strip element average value of each row are calculated respectively based on each row, so that the determined gap density and the strip element average value can have a high matching degree with the parallel strips.
[0201] Based on the above determination of the void density and the average value of the strip elements in each row in units of rows, the key result information can be determined in the following manner.
[0202] Optionally, an optional manner of the above step S802 includes:
[0203] The product of the void density and the average value of the strip elements of each row is calculated respectively to obtain the representation degree value of each row; each element of the row with the largest representation degree value is used as the above key result information.
[0204] Optionally, in the present application, the gap density and the strip element average value determined by the above-mentioned embodiment are both positively correlated with the key result information. Specifically, for the original image, there are inevitably some interference information between the real two parallel strips of the object, such as weeds between two cotton rows, which belong to interference information. Obviously, the more interference information there is, the smaller the gap density between the parallel strips of the object will be. On the contrary, the less interference information there is, the greater the gap density between the parallel strips of the object will be. Based on this, the row with a larger gap density can be found in each row for subsequent calculations, so as to minimize the influence of interference information on subsequent calculations. If the average value of the strip elements of a row is larger, it indicates that the more elements that can characterize the straight line in the row, the greater the possibility of forming parallel strips. Therefore, the gap density and the average value of the strip elements are combined. The gap density and the strip element average value of each row are multiplied to obtain the representation degree value of each row. The larger the representation degree value of a row is, the more the row can be used to accurately identify the parallel strips in the original image. On this basis, each element in the row with the largest representation degree value can be used as key result information. That is, the key result information includes a row of elements in the two-dimensional array.
[0205] In this embodiment, by multiplying the gap density and the average value of the strip elements that are positively correlated with the key result information, the representation degree value of each row can be obtained, and then the elements of the row with the largest representation degree value can be used as the key result information, so that the key result information can be used to accurately identify parallel strips in the original image.
[0206] As an optional implementation, after obtaining the key result information, the key result information may be filtered in the following manner to remove interference information and improve the availability of learning samples.
[0207] Fig.10 A flowchart for filtering key result information is shown below. Fig.10 As shown, after determining the key result information, it also includes:
[0208] S1001. Obtain at least one continuous element sequence according to the element value of each element in the row where the key result information is located.
[0209] Optionally, at least one continuous element sequence can be obtained based on the element value of each element of the row with the largest representation degree value. Among them, the continuous element sequence is a sequence composed of continuous strip elements, and the elements in the continuous element sequence are all strip elements. Specifically, the elements in the row with the largest representation degree value can be traversed, and whenever the element value of a traversed element is 0 or less than a threshold, that is, a non-strip element is traversed, the strip element before the traversed element and the element between the first strip element after the previous continuous element sequence are taken as a continuous element sequence, and so on, until the row with the largest representation degree value is traversed.
[0210] For example, assuming that the elements of the row with the largest representation degree value are arranged in order: 35600653405326, where the value of 0 indicates that the element at that location is a non-strip element, and the non-zero value indicates that the element at that location is a strip element, then three consecutive element sequences can be obtained through the above process, namely: 356, 6534, and 5326.
[0211] S1002 . Select a threshold value according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold value of the continuous element sequence.
[0212] By analyzing the element values of each element in the continuous element sequence, a suitable initial denoising threshold can be selected. For example, if the element values of most elements in the continuous element sequence are large, a large threshold can be selected as the initial denoising threshold.
[0213] S1003: Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filter the elements in the continuous element sequence to obtain denoised key result information.
[0214] In the first way, when filtering elements in a continuous element sequence based on an initial denoising threshold, the element value of an element whose element value is less than the initial denoising threshold may be adjusted, for example, the element value may be adjusted to 0, so that the element becomes a non-band element.
[0215] In the second method, the initial denoising threshold can be multiplied by a preset coefficient to obtain a target denoising threshold. The target denoising threshold is used to filter the continuous element sequence. Specifically, the element value of the element whose element value is less than the target denoising threshold can be adjusted, for example, the element value is adjusted to 0, so that the element becomes a non-band element. Since the preset coefficient is integrated into the initial denoising threshold, the filtering process of the continuous element sequence can be made more flexible and controllable.
[0216] The present application embodiment further provides a method for extracting object information. Fig.11 A flow chart of the object information extraction method provided in the embodiment of the present application is shown as follows: Fig.11 As shown, the method includes:
[0217] S1101 . Perform image transformation on an image to be identified to obtain a transformed image, where the transformed image is a binarized image or a mask image of the image to be identified.
[0218] The specific method of performing image transformation on the image to be recognized can refer to the aforementioned step S402, which will not be described in detail here.
[0219] S1102, performing Hough line detection on the transformed image to obtain a line detection result.
[0220] The execution method of this step is the same as that of the aforementioned step S403. Please refer to the description of the aforementioned step S403 and will not be repeated here.
[0221] S1103, inputting the above straight line detection result into a pre-trained information extraction model to obtain target object information in the above image to be identified, where the target object information includes: position information of the target object in the above image to be identified.
[0222] The above information extraction model is trained based on the method of the above-mentioned embodiment.
[0223] The target object mentioned above refers to a parallel strip of an object in the image to be identified, such as a cotton row, an electric wire or an electric pole.
[0224] As can be seen from the above embodiments, the position information identified by the information extraction model can be key result information, or can be angle and width. Based on the position information, the position of the parallel strips in the image to be identified can be known.
[0225] Fig.12 This is a visualization diagram of the location information obtained based on the information extraction model, such as Fig.12 As shown, each line segment corresponds to an object parallel strip in the original image, the height of the line segment in the vertical direction represents the angle of the object parallel strip, and the length of the line segment in the horizontal direction represents the width of the object parallel strip.
[0226] In this embodiment, after the straight line detection result is input into the information extraction model, the position information of the target object in the image to be identified can be obtained. Since the information extraction model has the ability to identify parallel strips, the complexity of the features of parallel strips is lower than that of the features of the object itself. Therefore, the complexity of the calculation can be reduced and the applicability can be greatly improved.
[0227] Based on the same inventive concept, an information extraction model training device corresponding to the information extraction model training method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned information extraction model training method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0228] Fig.13 The module structure diagram of the information extraction model training device provided in the embodiment of the present application is as follows: Fig.13 As shown, the device comprises:
[0229] The training module 1301 is used to iteratively train the initial extraction model using the training sample set until a preset iteration stop condition is met; wherein the training sample set includes learning samples represented by straight line detection results, and label information obtained by annotation based on the straight line detection results; the learning samples are obtained by performing Hough line detection on the original image, and the original image includes a linear or strip-shaped target object; the label information is used to represent the actual position of the target object in the original image.
[0230] The determination module 1302 is used to determine the current initial extraction model as the information extraction model when the initial extraction model meets a preset iteration stop condition.
[0231] As an optional implementation, the training module 1301 is specifically used for:
[0232] The learning sample is input into the current initial extraction model to obtain the current prediction result output by the current initial extraction model based on the learning sample.
[0233] The current loss function is constructed based on the current prediction result, current label information and preset sample weights.
[0234] The current loss function is used to back-propagate the current initial extraction model to update the model parameters of the current initial extraction model.
[0235] As an optional implementation, the training module 1301 is specifically used for:
[0236] Determine the intersection-over-union ratio of the current label information and the current prediction result.
[0237] The sample weight is used as an influencing factor to determine the cross entropy between the current label information and the current prediction result.
[0238] Based on the intersection-over-union ratio and the cross entropy, a current loss function is constructed.
[0239] As an optional implementation, the training module 1301 is further used for:
[0240] Get multiple original images.
[0241] The plurality of original images are respectively subjected to image transformation to obtain a plurality of transformed images, wherein the transformed images are binary images or mask images of the original images.
[0242] Performing Hough line detection on the multiple transformed images respectively to obtain multiple line detection results, and using the multiple line detection results as the learning samples; wherein the line detection results include multiple rows of elements, each element corresponds to a straight line in the original image, the position of each element is used to represent the position of the corresponding straight line in the original image, and the element value of each element is used to represent the reliability of the straight line.
[0243] According to the element values of the multiple rows of elements, key result information for characterizing the real position of the target object in the original image is determined, and the key result information is used as the label information.
[0244] As an optional implementation, the training module 1301 is specifically used for:
[0245] A plurality of key factor information is determined according to the element values of the plurality of rows of elements, wherein the key factor information includes: gap density and strip element average value.
[0246] The key result information is determined according to the plurality of key factor information.
[0247] As an optional implementation, the multiple rows of elements are stored in a two-dimensional array, the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines in the Hough space passing through the point corresponding to the position in the two-dimensional array.
[0248] The training module 1301 is specifically used for:
[0249] According to the element values of the elements in each row of the two-dimensional array, the stripe elements and the non-stripe elements in each row are determined.
[0250] The ratio of the number of non-strip elements in each row to the sum of the number of strip elements and non-strip elements in each row is calculated to obtain the void density of each row.
[0251] The average values of the element values of the strip elements in each row are calculated respectively to obtain the average value of the strip elements in each row.
[0252] As an optional implementation, the training module 1301 is specifically used for:
[0253] The product of the void density of each row and the average value of the strip elements is calculated to obtain the characterization degree value of each row.
[0254] Each element of the row with the largest representation degree value is used as the key result information.
[0255] As an optional implementation, the training module 1301 is specifically used for:
[0256] At least one continuous element sequence is obtained according to the element value of each element in the row where the key result information is located.
[0257] A threshold is selected according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold of the continuous element sequence.
[0258] Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filtering processing is performed on the elements in the continuous element sequence to obtain denoised key result information.
[0259] As an optional implementation, the training module 1301 is specifically used for:
[0260] Taking the center point of each transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the line detection result.
[0261] As an optional implementation, there is at least one hole convolution layer in the initial extraction model; and / or, in the initial extraction model, the initialization convolution parameter of the last layer is a preset value.
[0262] As an optional implementation, the convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
[0263] As an optional implementation, the original image includes any one of the following: an image containing crop rows, an image of electric poles, and an image of electric wires.
[0264] The recognition result obtained by the information extraction model processing the original image containing the target object includes: the position information of the target object in the original image.
[0265] Based on the same inventive concept, an object information extraction device corresponding to the object information extraction method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned object information extraction method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0266] Fig.14The module structure diagram of the object information extraction device provided in the embodiment of the present application is as follows: Fig.14 As shown, the device comprises:
[0267] The transformation module 1401 is used to perform image transformation on the image to be identified to obtain a transformed image, where the transformed image is a binary image or a mask image of the image to be identified.
[0268] The detection module 1402 is used to perform Hough line detection on the transformed image to obtain a line detection result.
[0269] The extraction module 1403 is used to input the straight line detection result into a pre-trained information extraction model to obtain the target object information in the image to be identified; the target object information includes: the position information of the target object in the image to be identified.
[0270] The information extraction model is trained based on the method described in the above embodiment.
[0271] The present application also provides a smart agriculture system, which includes:
[0272] The acquisition device is used to acquire the original image.
[0273] A processing device is used to process the original image obtained by the acquisition device according to the object information extraction method described in the aforementioned method embodiment to obtain the target object information in the original image.
[0274] A graphical interface device is used to generate display information of the corresponding original object according to the target object information.
[0275] It should be noted that the smart agriculture system can execute the method steps in the aforementioned method embodiment. In addition to the technical effects that can be achieved by the aforementioned method, it can facilitate users to view the target object information in the plot area corresponding to the original image. For example, it is convenient for users to clearly view information such as crop rows, wires and / or poles. It is also convenient for the system to control the operating equipment to perform adaptive operations based on the target object information, or to send the target object information to the operating equipment to enable the operating equipment to operate autonomously. For example, the operating equipment can accurately operate along the crop rows and avoid obstacles such as wires or poles.
[0276] The present application embodiment also provides an electronic device 150, such as Fig.15 As shown, it is a schematic diagram of the structure of the electronic device 150 provided in the embodiment of the present application, including: a processor 1501, a memory 1502, and optionally, a bus 1503. The memory 1502 stores machine-readable instructions executable by the processor 1501 (for example, Fig.13The training module 1301 and the determination module 1302 in the device are executed in accordance with the instructions, or, Fig.14 In the device, the transformation module 1401, the detection module 1402, the extraction module 1403 corresponding to the execution instructions, etc.), when the electronic device 150 is running, the processor 1501 communicates with the memory 1502 through the bus 1503, and the machine-readable instructions are executed by the processor 1501 to perform the method steps in the aforementioned method embodiment.
[0277] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned object information extraction method are executed.
[0278] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0279] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.
[0280] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.
Claims
1. A method for training an information extraction model, characterized in that: include: The initial extraction model is iteratively trained using a training sample set until a preset iteration stop condition is met; wherein the training sample set includes a learning sample represented by a straight line detection result and label information obtained by annotating the straight line detection result; the learning sample is obtained by performing Hough line detection on an original image, and the original image includes a linear or strip-shaped target object; the label information is used to represent the real position of the target object in the original image; When the initial extraction model meets a preset iteration stop condition, the current initial extraction model is determined as the information extraction model.
2. The method according to claim 1, characterized in that The iterative training of the initial extraction model using the training sample set includes: Inputting a learning sample into the current initial extraction model to obtain a current prediction result output by the current initial extraction model based on the learning sample; The current loss function is constructed based on the current prediction result, the current label information and the preset sample weights; The current loss function is used to back-propagate the current initial extraction model to update the model parameters of the current initial extraction model.
3. The method according to claim 2, characterized in that The current loss function is constructed based on the current prediction result, the current label information and the preset sample weight, including: Determine the intersection-over-union ratio of the current label information and the current prediction result; Using the sample weight as an influencing factor, determining the cross entropy between the current label information and the current prediction result; Based on the intersection-over-union ratio and the cross entropy, a current loss function is constructed.
4. The method according to claim 1, characterized in that: The training sample set is obtained by the following steps: Get multiple original images; Performing image transformation on the multiple original images respectively to obtain multiple transformed images, where the transformed images are binary images or mask images of the original images; Performing Hough line detection on the multiple transformed images respectively to obtain multiple line detection results, and using the multiple line detection results as the learning samples; wherein the line detection results include multiple rows of elements, each element corresponds to a straight line in the original image, the position of each element is used to represent the position of the corresponding straight line in the original image, and the element value of each element is used to represent the reliability of the straight line; According to the element values of the multiple rows of elements, key result information for characterizing the real position of the target object in the original image is determined, and the key result information is used as the label information.
5. The method according to claim 4, characterized in that The determining, according to the element values of the multiple rows of elements, key result information for characterizing the real position of the target object in the original image comprises: Determine multiple key factor information according to the element values of the multiple rows of elements, the key factor information including: gap density and strip element average value; The key result information is determined according to the plurality of key factor information.
6. The method according to claim 5, characterized in that The multiple rows of elements are stored in a two-dimensional array, the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines passing through the point corresponding to the position in the two-dimensional array in the Hough space; The determining of multiple key factor information according to the element values of the multiple rows of elements includes: Determine the stripe elements and the non-stripe elements in each row according to the element values of the elements in each row of the two-dimensional array; Calculate the ratio of the number of non-strip elements in each row to the sum of the number of strip elements and non-strip elements in each row to obtain the void density of each row; The average values of the element values of the strip elements in each row are calculated respectively to obtain the average value of the strip elements in each row.
7. The method according to claim 6, characterized in that The determining the key result information according to the plurality of key factor information includes: Calculate the product of the void density of each row and the average value of the strip elements to obtain the representation degree value of each row; Each element of the row with the largest representation degree value is used as the key result information.
8. The method according to claim 7, characterized in that After determining the key result information, the method further includes: Obtain at least one continuous element sequence according to the element value of each element in the row where the key result information is located; Selecting a threshold value according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold value of the continuous element sequence; Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filtering processing is performed on the elements in the continuous element sequence to obtain denoised key result information.
9. The method according to claim 4, characterized in that The Hough line detection is performed on the multiple transformed images respectively to obtain multiple line detection results, including: Taking the center point of each transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the line detection result.
10. The method according to claim 1, characterized in that In the initial extraction model, there is at least one hole convolution layer; and / or, in the initial extraction model, the initialization convolution parameter of the last layer is a preset value.
11. The method according to claim 10, characterized in that The convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
12. The method according to any one of claims 1 to 11, characterized in that: The original image includes any one of the following: an image containing crop rows, an image of electric poles, an image of electric wires; The recognition result obtained by the information extraction model processing the original image containing the target object includes: the position information of the target object in the original image.
13. A method for extracting object information, characterized in that: include: Performing image transformation on the image to be identified to obtain a transformed image, wherein the transformed image is a binary image or a mask image of the image to be identified; Performing Hough line detection on the transformed image to obtain a line detection result; Inputting the straight line detection result into a pre-trained information extraction model to obtain target object information in the image to be identified; The target object information includes: position information of the target object in the image to be identified, wherein the information extraction model is trained based on the method described in any one of claims 1-12.
14. An information extraction model training device, characterized in that: include: A training module, used for iteratively training the initial extraction model using a training sample set until a preset iteration stop condition is met; wherein the training sample set includes a learning sample represented by a straight line detection result, and label information obtained by annotating based on the straight line detection result; the learning sample is obtained by performing Hough line detection on an original image, and the original image includes a linear or strip-shaped target object; the label information is used to represent the real position of the target object in the original image; The determination module is used to determine the current initial extraction model as the information extraction model when the initial extraction model meets a preset iteration stop condition.
15. An object information extraction device, characterized in that: include: A transformation module, used for performing image transformation on the image to be identified to obtain a transformed image, wherein the transformed image is a binary image or a mask image of the image to be identified; A detection module, used for performing Hough line detection on the transformed image to obtain a line detection result; An extraction module, used for inputting the straight line detection result into a pre-trained information extraction model to obtain target object information in the image to be identified; The target object information includes: position information of the target object in the image to be identified, wherein the information extraction model is trained based on the method described in any one of claims 1-12.
16. A smart agricultural system, characterized in that: include: An acquisition device, used for acquiring an original image; A processing device, configured to process the original image obtained by the acquisition device according to the object information extraction method according to claim 13, so as to obtain target object information in the original image; A graphical interface device is used to generate display information of the corresponding original object according to the target object information.
17. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to execute the steps of the information extraction model training method as described in any one of claims 1 to 12 or the steps of the object information extraction method as described in claim 13.
18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the information extraction model training method according to any one of claims 1 to 12 or the steps of the object information extraction method according to claim 13.
Citation Information
Patent Citations
Tree space-based three-dimensional tree model matching method
CN112328821A
Method and device for identifying farmland soil surface, agricultural machine and medium
CN113807131A
Position detecting method
JP1995168940A
Adaptive pattern recognition based control system and method
US10361802B1
Artificial intelligence using convolutional neural network with hough transform
US20220122267A1
