Target object recognition method and device, electronic equipment and storage medium

Abstract

The application relates to the field of data recognition, and discloses a target object recognition method and device, electronic equipment and a storage medium. The method comprises the following steps: inputting a to-be-recognized image containing a target object into a target recognition model to obtain a result image; the target recognition model comprises an encoder, a decomposer and a decoder; the encoder comprises multiple convolution layers, the decomposer comprises at least one decomposition layer, and the decoder comprises multiple deconvolution layers; each deconvolution layer is used for fusing a received image, a first feature map output by a convolution layer at a current level and a feature matrix output by a connected decomposition layer, obtaining a second feature map at the current level, and transmitting the second feature map at the current level to a next deconvolution layer. According to the scheme provided in the application, comprehensive analysis of multi-level features is realized through a target recognition model, more semantic features are obtained through feature decomposition and fusion, the recognition accuracy is improved, and the problem of low recognition accuracy of a traditional recognition model is solved.

Description

Technical Field

[0001] This application relates to the field of data recognition technology, and in particular to a method, apparatus, electronic device and storage medium for identifying a target object. Background Technology

[0002] In the field of data recognition, the identification of objects in images is often involved. Especially in autonomous driving scenarios, the accuracy of object recognition will directly affect the safety of autonomous driving.

[0003] In related technologies, the task of object recognition is generally achieved through machine learning algorithms, such as traditional recognition models built on deep learning networks such as CNN (Convolutional Neural Networks) and FCN (Fully Convolutional Networks). However, due to the inherent structural limitations of these deep learning networks, the accuracy of object recognition in images is low, and the recognition effect is poor for smaller objects in images, making it difficult to meet the higher accuracy data recognition requirements in practical applications. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a method, apparatus, electronic device, and storage medium for identifying targets, which can improve the identification accuracy for smaller targets by structural improvements to the target identification model.

[0005] The first aspect of this application provides a method for identifying a target object, the method comprising:

[0006] The image to be identified, containing the target object, is input into the target recognition model to obtain the result image output by the target recognition model;

[0007] The resulting image includes the recognition results of marking the target objects in the image to be recognized;

[0008] The target recognition model includes an encoder, a decomposer, and a decoder;

[0009] The encoder includes multiple convolutional layers. Each convolutional layer is used to extract the semantic features of the received image at the current layer and output a first feature map containing the semantic features to the next convolutional layer connected to the current convolutional layer and the decoder.

[0010] The decomposer includes at least one decomposition layer, and the decoder includes multiple deconvolution layers. Each decomposition layer is connected to a convolution layer and a deconvolution layer, respectively. Each decomposition layer is used to perform feature decomposition on the received first feature map to obtain a feature matrix, and output the feature matrix to the deconvolution layer connected to the current decomposition layer.

[0011] Each deconvolutional layer is used to fuse the received image, the first feature map output by the convolutional layer corresponding to the current layer, and the feature matrix output by the connected decomposition layer to obtain the second feature map of the current layer, and then pass the second feature map of the current layer to the next deconvolutional layer connected to the current deconvolutional layer.

[0012] According to the target object identification method provided in this application, each decomposition layer specifically performs feature decomposition on the received first feature map through the following process:

[0013] Determine the image matrix corresponding to the received first feature map;

[0014] Perform singular value decomposition on the image matrix to obtain the singular value decomposition result;

[0015] Based on the singular value decomposition results, the feature matrix corresponding to the first feature map is determined.

[0016] According to the target object identification method provided in this application, the step of performing singular value decomposition on the image matrix to obtain the singular value decomposition result includes:

[0017] The transpose of the image matrix is ​​determined, and based on the image matrix and its transpose, the left singular matrix and right singular matrix corresponding to the image matrix are determined respectively; wherein, the left singular matrix includes multiple left singular vectors, and the right singular matrix includes multiple right singular vectors;

[0018] Based on the image matrix, the plurality of left singular vectors, and the plurality of right singular vectors, determine the singular value matrix corresponding to the image matrix;

[0019] The left singular matrix, the right singular matrix, and the singular value matrix are taken as the singular value decomposition result.

[0020] According to the target object identification method provided in this application, determining the feature matrix corresponding to the first feature map based on the singular value decomposition result includes:

[0021] Based on at least one of the left singular matrix, right singular matrix, and singular value matrix in the singular value decomposition result, the feature matrix corresponding to the first feature map is determined.

[0022] According to the target object recognition method provided in this application, the left singular matrix corresponding to the image matrix is ​​determined based on the image matrix and the transpose of the image matrix, including:

[0023] Multiplying the image matrix by its transpose yields the first matrix;

[0024] Eigenvalue decomposition is performed on the first matrix to obtain multiple left singular vectors;

[0025] The left singular vectors are concatenated to obtain the left singular matrix.

[0026] According to the target object identification method provided in this application, the right singular matrix corresponding to the image matrix is ​​determined based on the image matrix and the transpose of the image matrix, including:

[0027] Multiplying the transpose of the image matrix by the image matrix yields the second matrix;

[0028] Eigenvalue decomposition of the second matrix yields multiple right singular vectors;

[0029] The right singular vectors are concatenated to obtain the right singular matrix.

[0030] According to the target object recognition method provided in this application, the deconvolution layer specifically fuses the received image, the first feature map output by the convolutional layer corresponding to the current level, and the feature matrix output by the connected decomposition layer through the following process:

[0031] Determine the image matrix of the received images to obtain the third matrix;

[0032] Determine the image matrix of the first feature map output by the convolutional layer corresponding to the current level to obtain the fourth matrix;

[0033] The third matrix, the fourth matrix, and the feature matrix output by the connected decomposition layer are fused together.

[0034] A second aspect of this application provides a target identification device, the device comprising:

[0035] The recognition module is used to input the image to be recognized containing the target object into the target recognition model and obtain the result image output by the target recognition model;

[0036] The resulting image includes the recognition results of marking the target objects in the image to be recognized;

[0037] The target recognition model includes an encoder, a decomposer, and a decoder;

[0038] The encoder includes multiple convolutional layers. Each convolutional layer is used to extract the semantic features of the received image at the current layer and output a first feature map containing the semantic features to the next convolutional layer connected to the current convolutional layer and the decoder.

[0039] The decomposer includes at least one decomposition layer, and the decoder includes multiple deconvolution layers. Each decomposition layer is connected to a convolution layer and a deconvolution layer respectively. Each decomposition layer is used to perform feature decomposition on the received first feature map to obtain a feature matrix, and output the feature matrix to the deconvolution layer corresponding to the current decomposition layer.

[0040] Each deconvolutional layer is used to fuse the received image, the first feature map output by the convolutional layer corresponding to the current layer, and the feature matrix output by the connected decomposition layer to obtain the second feature map of the current layer, and then pass the second feature map of the current layer to the next deconvolutional layer connected to the current deconvolutional layer.

[0041] A third aspect of this application provides an electronic device, comprising:

[0042] Processor; and

[0043] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0044] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0045] The technical solution provided in this application may include the following beneficial effects:

[0046] Because the encoder in the target recognition model can extract semantic features from multiple levels using multiple convolutional layers, the decomposer can decompose the first feature map to obtain a feature matrix, and the decoder can use multiple deconvolutional layers to fuse the semantic features of the corresponding levels with the feature matrix output by the connected decomposition layers, the target recognition model can achieve comprehensive analysis of multi-level features in the image to be recognized, and can obtain more semantic features through feature decomposition and fusion. It can fully obtain the semantic features in the image to be recognized, and thus can more accurately recognize small targets in the image to be recognized, improving the recognition accuracy of the target.

[0047] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0048] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0049] Figure 1 This is a schematic flowchart illustrating the target object identification method shown in the embodiments of this application;

[0050] Figure 2 This is a schematic diagram of the structure of the improved target recognition model shown in the embodiments of this application;

[0051] Figure 3 This is a schematic diagram of a road image including road signs in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of a road sign label corresponding to a road image containing a road sign in an embodiment of this application;

[0053] Figure 5 This is one of the schematic diagrams of road sign recognition results obtained by a traditional recognition model in the embodiments of this application;

[0054] Figure 6 This is the second schematic diagram of the road sign recognition result obtained by the traditional recognition model in the embodiments of this application;

[0055] Figure 7 This is one of the schematic diagrams of road sign recognition results obtained by the improved target recognition model in the embodiments of this application;

[0056] Figure 8 This is the second schematic diagram of the road sign recognition result obtained by the improved target recognition model in the embodiments of this application;

[0057] Figure 9 This is a schematic diagram of the target object identification device shown in the embodiments of this application;

[0058] Figure 10 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0059] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0060] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0061] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0062] This embodiment relates to the field of data recognition, and can be specifically applied to target object recognition scenarios, such as the recognition of target objects in the vehicle driving environment during autonomous driving. In related technologies, due to the unreasonable network structure of the machine learning model used for target object recognition, fewer semantic features can be extracted from the image to be recognized, resulting in the inability to effectively recognize small target objects in the image to be recognized, and the recognition accuracy is low.

[0063] To address the aforementioned issues, this application provides a target object identification method that improves the identification accuracy for small-sized targets by refining the target identification model structure.

[0064] The following combination Figures 1 to 10 This application describes in detail the technical solutions for the target object identification method, apparatus, electronic device, and storage medium in its embodiments.

[0065] Figure 1 This is a schematic flowchart illustrating the target object identification method shown in the embodiments of this application.

[0066] See Figure 1 The target object identification method provided in this application embodiment specifically includes:

[0067] Step 101: Input the image containing the target object into the target recognition model to obtain the result image output by the target recognition model;

[0068] The result image includes the recognition results of marking the target objects in the image to be recognized;

[0069] The target recognition model includes an encoder, a decomposer, and a decoder;

[0070] The encoder includes multiple convolutional layers. Each convolutional layer is used to extract the semantic features of the received image at the current layer and outputs a first feature map containing the semantic features to the next convolutional layer connected to the current convolutional layer and the decoder.

[0071] The decomposer includes at least one decomposition layer, and the decoder includes multiple deconvolution layers. Each decomposition layer is connected to a convolution layer and a deconvolution layer respectively. Each decomposition layer is used to perform feature decomposition on the received first feature map to obtain a feature matrix, and output the feature matrix to the deconvolution layer connected to the current decomposition layer.

[0072] Each deconvolutional layer is used to fuse the received image, the first feature map output by the current convolutional layer, and the feature matrix output by the connected decomposition layer to obtain the second feature map of the current layer, and then pass the second feature map of the current layer to the next deconvolutional layer connected to the current deconvolutional layer.

[0073] In this embodiment, the execution entity of the target object recognition method can be a processor or a server. For example, in an autonomous driving scenario, the execution entity of the target object recognition method can be a processor deployed on the vehicle, or a processor or server deployed outside the vehicle.

[0074] Understandably, the image to be identified can be acquired by image acquisition devices. For example, in autonomous driving scenarios, the image to be identified can be acquired by a camera installed on the vehicle, or it can be acquired by roadside testing devices installed on both sides of the road.

[0075] In this embodiment, the identification of the target object can be understood as locating the edge of the target object from the image to be identified, thereby determining the outline of the target object and extracting the target object from the image to be identified. The target object can be a marker that needs to be used in subsequent decision-making processes. For example, in the autonomous driving scenario, the target object can be a traffic light, lane line, etc.

[0076] The result image is an image with the target object's location marked on the image to be identified. The location of the target object can be determined from the result image, which is the target object identification result.

[0077] It should be noted that the target recognition model is obtained by training a machine learning model on the image samples to be recognized. It extracts semantic features of the target object from the image to be recognized through semantic segmentation, thereby realizing the recognition of the target object.

[0078] This embodiment improves the network structure of the target recognition model by building an encoder, decomposer and decoder to achieve semantic feature extraction at different levels. At the same time, it can extract the feature matrix based on the first feature map extracted by the convolutional layer, obtain more semantic features, and make feature extraction more complete. This results in higher recognition accuracy and better recognition effect for smaller targets in the image to be recognized.

[0079] In some embodiments, the encoder can be constructed by sequentially connecting multiple convolutional layers. In the encoder, the input of each convolutional layer is connected to the output of the previous convolutional layer, and the output of each convolutional layer is connected to the input of the next convolutional layer. This allows the semantic features extracted by each convolutional layer to be passed step by step until the last convolutional layer is reached, thereby realizing the function of acquiring multi-level semantic features in the image to be recognized.

[0080] In an exemplary embodiment, the encoder may specifically include a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, which are connected sequentially.

[0081] The first convolutional layer is used to extract the semantic features of the image to be recognized at the first level, and outputs the first feature map containing the semantic features of the first level to the second convolutional layer and the decoder respectively.

[0082] The second convolutional layer is used to extract the semantic features of the image to be recognized at the second level based on the semantic features of the first level, and outputs the first feature map containing the semantic features of the second level to the third convolutional layer and the decoder respectively.

[0083] The third convolutional layer is used to extract the semantic features of the image to be recognized at the third level based on the semantic features of the second level, and outputs the first feature map containing the semantic features of the third level to the fourth convolutional layer and the decoder respectively.

[0084] The fourth convolutional layer is used to extract the semantic features of the image to be recognized at the fourth level based on the semantic features of the third level, and outputs the first feature map containing the semantic features of the fourth level to the decoder.

[0085] This embodiment addresses the case where the encoder contains four convolutional layers. See [link to previous example]. Figure 2 The network architecture of the entire target recognition model can be roughly divided into four basic layers, namely Figure 2 The structure consists of four rows of layers from top to bottom. These four basic layers can be further divided into three parts: encoder 201, decoder 202, and decomposer 203. Encoder 201 and decoder 202 can be connected via a splicing channel. Figure 2The target recognition model shown has a kernel size of 3×3, padding of 1, and stride of 1.

[0086] like Figure 2 As shown, the encoder 201 has four convolutional layers connected sequentially inside. Each convolutional layer is defined according to the data transmission direction, that is, the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer correspond to the data transmission direction in sequence. Figure 2 The basic layers in encoder 201, from top to bottom, and the semantic features output by each convolutional layer in encoder 201 are transmitted through the first feature map that can represent the semantic information of the current layer.

[0087] On the one hand, each convolutional layer performs a downsampling operation on the input image, doubling the number of channels and reducing the image size (specifically, the length and width) to half. For example, if the input image size is 480×800, after one downsampling operation, the image size is reduced to 240×400. On the other hand, each convolutional layer performs convolution, normalization, and activation operations on the input image to extract the semantic features of the input image at the current level. This multi-layer convolutional layer enables the extraction of semantic features at multiple levels.

[0088] In some embodiments, the decomposer may specifically include multiple decomposition layers, with the input of each decomposition layer connected to the convolutional layer corresponding to the current layer, and the output of each decomposition layer connected to the deconvolutional layer corresponding to the next layer.

[0089] Each decomposition layer is used to perform feature decomposition on the first feature map of the current layer to obtain the feature matrix, and then input the feature matrix into the corresponding deconvolution layer of the next layer.

[0090] See Figure 2 In this embodiment, the decoder 202 and the encoder 201 have the same number of layers. Figure 2 In the scenario shown, decoder 202 contains four sequentially connected deconvolutional layers. These four deconvolutional layers are defined according to the data flow direction. Specifically, the first, second, third, and fourth deconvolutional layers are... Figure 2 The rows in the middle correspond to the basic layers from bottom to top within decoder 202.

[0091] See Figure 2 In this embodiment, three decomposition layers are set, which are respectively set at the first, second and third levels of the corresponding convolutional layers. That is, in this embodiment, the decomposer 203 specifically includes a first decomposition layer, a second decomposition layer and a third decomposition layer.

[0092] The input of the first decomposition layer is connected to the first convolutional layer, and the output of the first decomposition layer is connected to the third deconvolutional layer. The input of the second decomposition layer is connected to the second convolutional layer, and the output of the second decomposition layer is connected to the second deconvolutional layer. The input of the third decomposition layer is connected to the third convolutional layer, and the output of the third decomposition layer is connected to the first deconvolutional layer.

[0093] The first decomposition layer is used to perform feature decomposition on the first feature map of the first level to obtain the feature matrix of the first level, and then pass the feature matrix of the first level to the third deconvolution layer.

[0094] The second decomposition layer is used to perform feature decomposition on the first feature map of the second level to obtain the feature matrix of the second level, and then pass the feature matrix of the second level to the second deconvolution layer.

[0095] The third decomposition layer is used to perform feature decomposition on the first feature map of the third level to obtain the feature matrix of the third level, and then pass the feature matrix of the third level to the first deconvolution layer.

[0096] It should be noted that this embodiment mainly considers that the image size of the first feature map of the fourth level is small after multiple downsampling operations, so feature decomposition of the first feature map of this level is not required. Therefore, no decomposition layer is set for the convolutional layer of this level. If this target recognition method is applied to a scenario with higher recognition accuracy requirements, a decomposition layer can be set for the fourth convolutional layer, i.e., the fourth decomposition layer, to extract semantic features more fully. The number of decomposition layers in the decomposer can be reasonably set according to the actual application scenario requirements, which will not be elaborated on here.

[0097] In this embodiment, the decomposition layer can obtain richer semantic features based on the semantic features extracted by the convolutional layer, making the extracted semantic features more abundant.

[0098] In some embodiments, each decomposition layer may specifically perform feature decomposition on the received first feature map through the following process:

[0099] Determine the image matrix corresponding to the received first feature map;

[0100] Singular value decomposition is performed on the image matrix to obtain the singular value decomposition results;

[0101] Based on the singular value decomposition results, the feature matrix corresponding to the first feature map is determined.

[0102] In this embodiment, considering that the first feature map is essentially still an image, it can be represented in the form of an image matrix. For example, the pixel values ​​of each pixel in the first feature map can be matrix elements to construct an image matrix. In this case, the image matrix can represent the pixel information corresponding to the first feature map.

[0103] After constructing the image matrix, the feature matrix corresponding to the first feature map can be determined by the Singular Value Decomposition (SVD) algorithm. The SVD algorithm can transform the image matrix into multiple sub-matrices through matrix decomposition. The transformed sub-matrices can represent the feature information of the image matrix, so the feature matrix of the first feature map can be conveniently and accurately determined based on the result of the singular value decomposition.

[0104] In some embodiments, singular value decomposition is performed on the image matrix to obtain the singular value decomposition result, specifically including:

[0105] Determine the transpose of the image matrix. Based on the image matrix and its transpose, determine the corresponding left singular matrix and right singular matrix, respectively. The left singular matrix includes multiple left singular vectors, and the right singular matrix includes multiple right singular vectors.

[0106] Based on the image matrix, multiple left singular vectors, and multiple right singular vectors, determine the singular value matrix corresponding to the image matrix;

[0107] The left singular matrix, right singular matrix, and singular value matrix are taken as the results of singular value decomposition.

[0108] In this embodiment, the image matrix is ​​set to A, which is an m×n matrix. The singular value decomposition expression of the image matrix A can be expressed as follows:

[0109] (1)

[0110] Where A represents the image matrix, U represents the left singular matrix (an m × m matrix), Σ represents the singular value matrix (an m × n matrix where all elements except those on the main diagonal are 0, and each element on the main diagonal is called a singular value), and V represents the right singular matrix (an n × n matrix). This represents the transpose of a right singular matrix.

[0111] In practical applications, the left singular matrix U and the right singular matrix V can be derived from the image matrix A and the transpose of the image matrix. After solving for the left singular matrix U and the right singular matrix V, the singular value matrix Σ can be further determined by using multiple left singular vectors in the left singular matrix U, multiple right singular vectors in the right singular matrix V, and the image matrix, thus obtaining the singular value decomposition result.

[0112] In an exemplary embodiment, determining the left singular matrix corresponding to the image matrix based on the image matrix and its transpose specifically includes:

[0113] Multiplying the image matrix by its transpose yields the first matrix;

[0114] Eigenvalue decomposition of the first matrix yields multiple left singular vectors;

[0115] Multiple left singular vectors are concatenated to obtain a left singular matrix.

[0116] In this embodiment, when determining the left singular matrix, the image matrix A and the transpose of the image matrix are first... By performing matrix multiplication, we can obtain an m×m square matrix AA. T That is, the first matrix, since the first matrix AA T It is a square matrix, and square matrices can be decomposed into eigenvalues. Therefore, for the first matrix AA... T The eigenvalues ​​and eigenvectors (i.e., left singular vectors) obtained by eigenvalue decomposition satisfy the following expression:

[0117] (2)

[0118] Among them, AA T Describes the first matrix. Let i represent the i-th left singular vector. Let represent the eigenvalue corresponding to the i-th left singular vector.

[0119] After obtaining each left singular vector, concatenating all the left singular vectors will form an m×m left singular matrix U.

[0120] In an exemplary embodiment, determining the right singular matrix corresponding to the image matrix based on the image matrix and its transpose specifically includes:

[0121] Multiply the transpose of the image matrix by the image matrix to obtain the second matrix;

[0122] Eigenvalue decomposition of the second matrix yields multiple right singular vectors;

[0123] Multiple right singular vectors are concatenated to obtain a right singular matrix.

[0124] In this embodiment, when determining the right singular matrix, the transpose of the image matrix can be used. Performing matrix multiplication with the image matrix A yields an n×n square matrix A. T A, the second matrix, because the second matrix A T If A is a square matrix, and square matrices can be decomposed into eigenvalues, then for the second matrix A... T The eigenvalues ​​and eigenvectors (i.e., right singular vectors) obtained by performing eigenvalue decomposition on A satisfy the following expression:

[0125] (3)

[0126] Among them, A T A represents the second matrix. Let i represent the i-th right singular vector. Let represent the eigenvalue corresponding to the i-th right singular vector.

[0127] After obtaining each right singular vector, concatenating all the right singular vectors will form an n×n right singular matrix V.

[0128] After determining the left singular matrix U and the right singular matrix V, since all elements in the singular value matrix Σ except for the singular values ​​on the diagonal are 0, we can first solve for all the singular values ​​and then determine the singular value matrix Σ.

[0129] Each singular value can be solved using the following formula:

[0130] (4)

[0131] in, This represents the i-th singular value. Let represent the i-th eigenvalue. This eigenvalue can be the eigenvalue corresponding to the i-th left singular vector or the eigenvalue corresponding to the i-th right singular vector.

[0132] After determining all the singular values, each singular value is taken as an element on the diagonal of the singular value matrix Σ, and the remaining elements are set to 0, thus obtaining the singular value matrix Σ.

[0133] In some embodiments, the feature matrix corresponding to the first feature map is determined based on the singular value decomposition result, specifically including:

[0134] The feature matrix corresponding to the first feature map is determined based on at least one of the left singular matrix, right singular matrix, and singular value matrix in the singular value decomposition result.

[0135] In this embodiment, any one of the left singular matrix, right singular matrix, and singular value matrix can be used as the feature matrix corresponding to the first feature map. Alternatively, any of the left singular matrix, right singular matrix, and singular value matrix can be concatenated to obtain the feature matrix corresponding to the first feature map. This embodiment uses the method of directly using the left singular matrix as the feature matrix corresponding to the first feature to determine the feature matrix. This method can more conveniently determine the feature matrix corresponding to the first feature map.

[0136] It is understood that in this embodiment, the size ratio between the feature matrix obtained by the decomposition layer and the first feature map corresponding to the feature matrix is ​​1:2, thereby achieving data dimensionality reduction through singular value decomposition, which facilitates the rapid convergence of model parameters during the model training stage and can improve model training efficiency.

[0137] This embodiment can perform feature decomposition on the first feature map output by the convolutional layer by setting a decomposer, and then obtain other semantic features outside the convolutional layer by determining the feature matrix, resulting in richer semantic features.

[0138] In some embodiments, the decoder specifically includes multiple deconvolutional layers, which are connected sequentially.

[0139] Each deconvolutional layer is used to fuse the received image, the first feature map output by the current layer's corresponding convolutional layer, and the feature matrix output by the connected decomposition layer to obtain a second feature map containing the fused features of the current layer. The second feature map of the current layer is then passed to the next deconvolutional layer connected to the current deconvolutional layer, until the last deconvolutional layer outputs the result image.

[0140] It is understood that the image received by the deconvolution layer in this embodiment can be the second feature map of the previous level passed by the previous deconvolution layer connected to the current deconvolution layer, or it can be a feature map other than the first feature map output by the convolution layer corresponding to the current level. For example, it can be the first feature map output by the convolution layer of the last level, that is, the feature map output by the last downsampling.

[0141] In this embodiment, the decoder is constructed by sequentially connecting multiple deconvolutional layers. The decoder mainly realizes the fusion of semantic features in the first feature maps of different levels output by the encoder. On this basis, the feature matrix obtained by the decomposition layer can also be fused to more fully analyze the semantic features contained in the image to be recognized.

[0142] In this embodiment, deconvolutional layers correspond one-to-one with convolutional layers. Each deconvolutional layer corresponds to a convolutional layer at the corresponding level. The convolutional layer can output the first feature map of the current level to the deconvolutional layer corresponding to the current convolutional layer through the concatenation channel. At the same time, at least some deconvolutional layers can also be connected to at least one decomposition layer. In this embodiment, after receiving the first feature map of the current level, the decomposition layer extracts the feature matrix of the first feature map and can pass the extracted feature matrix to the deconvolutional layer corresponding to the next level. For example, taking the decomposition layer at the second level as an example, the decomposition layer can receive the first feature map output by the convolutional layer at the second level, extract the feature matrix, and then pass the feature matrix to the deconvolutional layer at the third level.

[0143] In an exemplary embodiment, where the encoder includes four convolutional layers, the decoder includes a first deconvolutional layer, a second deconvolutional layer, a third deconvolutional layer, and a fourth deconvolutional layer, which are connected sequentially.

[0144] The first deconvolutional layer is used to fuse the first feature map containing the semantic features of the fourth level output by the fourth convolutional layer in the encoder with the feature matrix output by the connected decomposition layer. The resulting second feature map containing the first level fused features will be passed to the second deconvolutional layer.

[0145] The second deconvolutional layer is used to fuse the second feature map output by the first deconvolutional layer, the first feature map containing the semantic features of the third level output by the third convolutional layer in the encoder, and the feature matrix output by the connected decomposition layer, and then pass the resulting second feature map containing the second level fused features to the third deconvolutional layer.

[0146] The third deconvolutional layer is used to fuse the second feature map output by the second deconvolutional layer, the first feature map containing the semantic features of the second level output by the second convolutional layer in the encoder, and the feature matrix output by the connected decomposition layer, and then pass the resulting second feature map containing the third level fused features to the fourth deconvolutional layer.

[0147] The fourth deconvolutional layer is used to fuse the second feature map output by the third deconvolutional layer with the first feature map output by the first convolutional layer in the encoder, which contains the semantic features of the first level, to obtain the result image.

[0148] In some embodiments, in order to more fully integrate and analyze semantic features at different levels, the first deconvolutional layer in this embodiment can also be connected to the input of the fourth convolutional layer through a splicing channel. This allows the first feature map of the third level input to the input of the fourth convolutional layer to be integrated during feature fusion analysis, resulting in better feature fusion performance and further improving the recognition accuracy of the target recognition model.

[0149] On the one hand, each deconvolution layer can perform an upsampling operation on the input feature map. The upsampling operation can reduce the number of channels of the input feature map by half and double the size of the feature map (specifically the length and width of the image). For example, if the size of the input feature map is 480×800, after performing an upsampling operation, the size of the image will increase to 960×1600.

[0150] On the other hand, deconvolutional layers can fuse the first feature map of the current layer output by the encoder, the feature map output by the previous connected deconvolutional layer, and the feature matrix output by the connected decomposition layer, thereby extracting more detailed semantic features.

[0151] In some embodiments, the deconvolutional layer can fuse the received image, the first feature map output by the convolutional layer corresponding to the current layer, and the feature matrix output by the connected decomposition layer through the following process:

[0152] Determine the image matrix of the received images to obtain the third matrix;

[0153] Determine the image matrix of the first feature map output by the convolutional layer corresponding to the current level to obtain the fourth matrix;

[0154] The third matrix, the fourth matrix, and the feature matrix output from the connected decomposition layer are fused together.

[0155] In this embodiment, the feature fusion process of the deconvolution layer can be understood as a matrix-level data fusion process, which can be achieved through matrix merging. In practical applications, matrix merging can be performed based on the actual input data of the deconvolution layer. For example, if the input data includes three parts: the feature map output by the previous deconvolution layer, the first feature map output by the current convolution layer, and the feature matrix output by the connected decomposition layer, the image matrix corresponding to the input feature map can be calculated separately. In this embodiment, the image matrix of the feature map output by the previous deconvolution layer is called the third matrix, and the image matrix of the first feature map output by the current convolution layer is called the fourth matrix. The third matrix, the fourth matrix, and the feature matrix output by the connected decomposition layer are then fused, which means that the third matrix, the fourth matrix, and the feature matrix can be merged. In practical applications, the matrix merging function can be implemented using the cat operation in PyTorch.

[0156] In this embodiment, the deconvolution layer in the decoder 202 can be used to fuse the received image, the first feature map output by the encoder 201, and the feature matrix output by the decomposition layer, thereby obtaining richer semantic features. Through the cooperation of the encoder 201, decoder 202, and decomposition layer 203, the target recognition model can output a more accurate result image.

[0157] In some embodiments, the target recognition model can be trained through the following process:

[0158] Based on the acquired image samples and label information, sample data is generated;

[0159] The sample data is divided into training data and test data, and the training data and test data are stored in the sample database;

[0160] Training and testing data are retrieved from the sample database. The pre-built target recognition network is trained using the training data, and the trained target recognition network is tested using the testing data to obtain the target recognition model.

[0161] It is understandable that an image sample refers to an image containing the target object to be identified. For example, when the target object is a road sign, the image sample can be an image containing the road sign. The label information refers to the marking information of the target object, specifically the marking information of the target object's edges, such as the marking information of the road sign.

[0162] This embodiment can use supervised training to train a target recognition model. After the target recognition model passes the test, it can be used in target recognition scenarios.

[0163] It should be noted that in this embodiment, after obtaining the training and test data during the training of the target recognition model, the training and test data are first stored. Specifically, they can be stored in a sample database, such as an MDB (Microsoft Database). When training the target recognition model later, the training and test data can be directly retrieved from the sample database. Specifically, the data read from the sample database can be parsed into a matrix and input into the edge detection network for training. For example, the MDB format data can be parsed into a 512×512×3 matrix and input into the target recognition network for training to obtain a trained target recognition model.

[0164] The model training method provided in this embodiment, compared with the method of reading and constructing training and test data on-site, can train the target recognition model more efficiently by storing and then retrieving the data, thus improving the efficiency of the target recognition model training process.

[0165] In some embodiments, before splitting the sample data into training data and test data, the following may also be included:

[0166] Once outlier data is identified in the sample data, the outlier data is corrected.

[0167] Among them, abnormal data refers to data with incorrect label information corresponding to image samples.

[0168] In this embodiment, before dividing the training data and test data, the sample data can be preprocessed. Specifically, abnormal data in the sample data can be extracted and corrected. For example, sample data whose label information does not correspond to the image sample due to incorrect labeling can be extracted. Such sample data whose label information does not correspond to the image sample is not standardized enough, and if it is directly used for training the target recognition model, it will affect the training accuracy of the target recognition model.

[0169] This embodiment uses the above preprocessing operations to modify non-standard label information, making the sample data more accurate and standardized, thereby providing accurate and reliable data for the training of the target recognition model.

[0170] More preferably, the target object recognition method provided in this application embodiment may further include, before inputting the image to be recognized containing the target object into the target recognition model:

[0171] Adjust the image size of the image to be recognized to the preset size.

[0172] Considering that the image sample size used by the target recognition model is relatively fixed during the training process, in order to obtain more accurate recognition results, the size of the image to be recognized can be adjusted first, and then the image to be recognized with the adjusted size can be input into the target recognition model, thereby improving the recognition accuracy of the target.

[0173] To verify the improved recognition accuracy of the target object recognition method provided in this embodiment, this embodiment uses a traditional recognition model and a target recognition model to recognize the same image to be recognized. In this embodiment, the traditional recognition model can be implemented based on the Canny operator.

[0174] In the comparative verification, the image to be identified is a road image that includes road signs. Figure 3 An exemplary image to be identified is shown, which is captured by a camera mounted on the front of a vehicle. This image includes vehicles on the road, road signs on both sides of the road, and trees. The identification objective of this embodiment is to identify the location of key road signs from the road image to be identified. Specifically, Figure 3 The image to be identified shows four road signs, see details below. Figure 3 The location enclosed in the rectangle indicates the road sign recognition label corresponding to the image to be recognized. (See also...) Figure 4 After identifying the target object in the image using a traditional recognition model, the road sign recognition results can be found in [reference needed]. Figure 5 and Figure 6 After the improved target recognition model provided in this embodiment performs target object recognition on the image to be recognized, the road sign recognition result can be found in [reference needed]. Figure 7 and Figure 8 .

[0175] Will Figure 5 and Figure 6 The road sign recognition results shown Figure 7 and Figure 8 The road sign recognition results shown are respectively with Figure 4 By comparing the road sign identification tags shown, it can be found that... Figure 5 and Figure 6 The road sign recognition results shown are poor for license plates at the furthest distance, with instances of false recognition. Figure 7 and Figure 8 The road sign recognition results shown still have a good recognition effect on the license plates that are furthest away.

[0176] Therefore, it can be seen that for road signs in the distance, due to their small size in the road image, traditional recognition models have difficulty accurately recognizing road signs with small size (e.g., less than ten pixels) in the road image. However, the target recognition model provided in this embodiment can extract semantic features in the road image more fully after structural improvement, and can still accurately recognize road signs with small size in the road image. Compared with traditional recognition models, the target recognition model provided in this embodiment has higher recognition accuracy.

[0177] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a target object identification device, an electronic device, and corresponding embodiments.

[0178] Figure 9 This is a schematic diagram of the target object identification device shown in the embodiments of this application.

[0179] See Figure 9 The target object identification device provided in this application embodiment specifically includes:

[0180] The recognition module 301 is used to input the image to be recognized containing the target object into the target recognition model and obtain the result image output by the target recognition model;

[0181] The result image includes the recognition results of marking the target objects in the image to be recognized;

[0182] The target recognition model includes an encoder, a decomposer, and a decoder;

[0183] The encoder includes multiple convolutional layers. Each convolutional layer is used to extract the semantic features of the received image at the current layer and outputs a first feature map containing the semantic features to the next convolutional layer connected to the current convolutional layer and the decoder.

[0184] The decomposer includes at least one decomposition layer, and the decoder includes multiple deconvolution layers. Each decomposition layer is connected to a convolution layer and a deconvolution layer respectively. Each decomposition layer is used to perform feature decomposition on the received first feature map to obtain a feature matrix, and output the feature matrix to the corresponding deconvolution layer.

[0185] Each deconvolutional layer is used to fuse the received image, the first feature map output by the current convolutional layer, and the feature matrix output by the connected decomposition layer to obtain the second feature map of the current layer, and then pass the second feature map of the current layer to the next deconvolutional layer connected to the current deconvolutional layer.

[0186] In some embodiments, in the recognition module 301, each decomposition layer can specifically perform feature decomposition on the received first feature map through the following process:

[0187] Determine the image matrix corresponding to the received first feature map;

[0188] Singular value decomposition is performed on the image matrix to obtain the singular value decomposition results;

[0189] Based on the singular value decomposition results, the feature matrix corresponding to the first feature map is determined.

[0190] In some embodiments, in the recognition module 301, each decomposition layer can specifically perform singular value decomposition on the image matrix through the following process to obtain the singular value decomposition result:

[0191] Determine the transpose of the image matrix. Based on the image matrix and its transpose, determine the corresponding left singular matrix and right singular matrix, respectively. The left singular matrix includes multiple left singular vectors, and the right singular matrix includes multiple right singular vectors.

[0192] Based on the image matrix, multiple left singular vectors, and multiple right singular vectors, determine the singular value matrix corresponding to the image matrix;

[0193] The left singular matrix, right singular matrix, and singular value matrix are taken as the results of singular value decomposition.

[0194] In some embodiments, in the identification module 301, each decomposition layer can specifically determine the feature matrix corresponding to the first feature map based on the singular value decomposition result through the following process:

[0195] The feature matrix corresponding to the first feature map is determined based on at least one of the left singular matrix, right singular matrix, and singular value matrix in the singular value decomposition result.

[0196] In some embodiments, in the recognition module 301, each decomposition layer can specifically determine the left singular matrix corresponding to the image matrix based on the image matrix and the transpose of the image matrix through the following process:

[0197] Multiplying the image matrix by its transpose yields the first matrix;

[0198] Eigenvalue decomposition of the first matrix yields multiple left singular vectors;

[0199] Multiple left singular vectors are concatenated to obtain a left singular matrix.

[0200] In some embodiments, in the recognition module 301, each decomposition layer can specifically determine the right singular matrix corresponding to the image matrix based on the image matrix and the transpose of the image matrix through the following process:

[0201] Multiply the transpose of the image matrix by the image matrix to obtain the second matrix;

[0202] Eigenvalue decomposition of the second matrix yields multiple right singular vectors;

[0203] Multiple right singular vectors are concatenated to obtain a right singular matrix.

[0204] In some embodiments, in the recognition module 301, the deconvolution layer specifically fuses the received image, the first feature map output by the convolutional layer corresponding to the current level, and the feature matrix output by the connected decomposition layer through the following process:

[0205] Determine the image matrix of the received images to obtain the third matrix;

[0206] Determine the image matrix of the first feature map output by the convolutional layer corresponding to the current level to obtain the fourth matrix;

[0207] The third matrix, the fourth matrix, and the feature matrix output from the connected decomposition layer are fused together.

[0208] In summary, the target object recognition device provided in this application embodiment can achieve comprehensive analysis of multi-level features in the image to be recognized by the target recognition model, and can obtain more semantic features through feature decomposition and fusion. It can fully obtain the semantic features in the image to be recognized, and thus can more accurately recognize small target objects in the image to be recognized, thereby improving the recognition accuracy of the target object.

[0209] Figure 10 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0210] See Figure 10 The electronic device 400 includes a memory 401 and a processor 402.

[0211] Processor 402 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0212] Memory 401 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 402 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical discs, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation.

[0213] Furthermore, memory 401 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical discs may also be employed. In some embodiments, memory 401 may include removable storage devices that are readable and / or writable, such as laser discs (CDs), read-only digital versatile optical discs (e.g., DVD-ROMs, dual-layer DVD-ROMs), read-only Blu-ray discs, ultra-high density optical discs, flash memory cards (e.g., SD cards, mini SD cards, Micro-SD cards, etc.), magnetic floppy disks, etc. Computer-readable storage media do not contain carrier waves and transient electronic signals transmitted wirelessly or via wired connections.

[0214] The memory 401 stores executable code, which, when processed by the processor 402, can cause the processor 402 to execute part or all of the methods described above.

[0215] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0216] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0217] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of identifying an object, characterized by, The method comprises: inputting an image to be identified containing a target object into a target identification model to obtain a result image output by the target identification model; wherein the result image comprises an identification result of marking the target object in the image to be identified; the target identification model comprises an encoder, a decomposer and a decoder; the encoder comprises multiple convolution layers, each convolution layer is used to extract semantic features of a received image at a current level, and output a first feature map containing the semantic features to a next convolution layer connected with the current convolution layer and the decoder; the decomposer comprises multiple decomposition layers, and the decoder comprises multiple deconvolution layers, each decomposition layer is connected with a convolution layer and a deconvolution layer, each decomposition layer is used to perform feature decomposition on the received first feature map to obtain a feature matrix, and output the feature matrix to a deconvolution layer connected with the current decomposition layer; the input end of each decomposition layer is connected to the convolution layer corresponding to the current level, and the output end of each decomposition layer is connected to the deconvolution layer corresponding to the next level; each decomposition layer is used to perform feature decomposition on the first feature map of the current level to obtain the feature matrix, and input the feature matrix to the deconvolution layer corresponding to the next level; each deconvolution layer is used to fuse the received image, the first feature map output by the convolution layer corresponding to the current level, and the feature matrix output by the connected decomposition layer to obtain a second feature map of the current level, and pass the second feature map of the current level to the next deconvolution layer connected with the current deconvolution layer.

2. The method of claim 1, wherein Each decomposition layer specifically performs feature decomposition on the received first feature map through the following process: determine the image matrix corresponding to the received first feature map; perform singular value decomposition on the image matrix to obtain a singular value decomposition result; determine the feature matrix corresponding to the first feature map according to the singular value decomposition result.

3. The method of claim 2, wherein The singular value decomposition on the image matrix to obtain a singular value decomposition result comprises: determine the transpose of the image matrix, and determine the left singular matrix and the right singular matrix corresponding to the image matrix according to the image matrix and the transpose of the image matrix; wherein the left singular matrix comprises a plurality of left singular vectors, and the right singular matrix comprises a plurality of right singular vectors; determine the singular value matrix corresponding to the image matrix according to the image matrix, the plurality of left singular vectors and the plurality of right singular vectors; take the left singular matrix, the right singular matrix and the singular value matrix as the singular value decomposition result.

4. The method of claim 3, wherein The determination of the feature matrix corresponding to the first feature map according to at least one of the left singular matrix, the right singular matrix and the singular value matrix in the singular value decomposition result comprises: determine the feature matrix corresponding to the first feature map according to at least one of the left singular matrix, the right singular matrix and the singular value matrix in the singular value decomposition result.

5. The method of claim 3, wherein the step of identifying the target object is performed by using a neural network. The determination of the left singular matrix corresponding to the image matrix according to the image matrix and the transpose of the image matrix comprises: multiply the image matrix by the transpose of the image matrix to obtain a first matrix; performing eigen decomposition on the first matrix to obtain the plurality of left singular vectors; concatenate the plurality of left singular vectors to obtain the left singular matrix.

6. The method of claim 3, wherein the step of identifying the target object is performed by using a neural network. determining a right singular matrix corresponding to the image matrix according to the image matrix and a transpose of the image matrix, including: multiplying the transpose of the image matrix with the image matrix to obtain a second matrix; performing eigen decomposition on the second matrix to obtain the plurality of right singular vectors; concatenate the plurality of right singular vectors to obtain the right singular matrix.

7. The object recognition method according to claim 1, wherein The deconvolution layer specifically fuses the received image, the first feature map output by the convolution layer corresponding to the current level, and the feature matrix output by the connected decomposition layer through the following process: determining an image matrix of the received image to obtain a third matrix; determining an image matrix of the first feature map output by the convolution layer corresponding to the current level to obtain a fourth matrix; fusing the third matrix, the fourth matrix, and the feature matrix output by the connected decomposition layer.

8. An apparatus for identifying an object, characterized by including: a recognition module configured to input a to-be-recognized image containing a target object into a target recognition model to obtain a result image output by the target recognition model; wherein the result image includes a recognition result of marking the target object in the to-be-recognized image; the target recognition model includes an encoder, a decomposer, and a decoder; the encoder includes a plurality of convolution layers, each convolution layer being configured to extract a semantic feature of a received image at a current level and output a first feature map containing the semantic feature to a next convolution layer connected with the current convolution layer and the decoder; the decomposer includes a plurality of decomposition layers, and the decoder includes a plurality of deconvolution layers, each decomposition layer being connected with a convolution layer and a deconvolution layer, each decomposition layer being configured to perform eigen decomposition on the received first feature map to obtain a feature matrix and output the feature matrix to a deconvolution layer connected with the current decomposition layer, an input end of each decomposition layer being connected to the convolution layer corresponding to the current level, and an output end of each decomposition layer being connected to the deconvolution layer corresponding to a next level, each decomposition layer being configured to perform eigen decomposition on the first feature map of the current level to obtain the feature matrix and input the feature matrix to the deconvolution layer corresponding to the next level; each deconvolution layer is configured to fuse the received image, the first feature map output by the convolution layer corresponding to the current level, and the feature matrix output by the connected decomposition layer to obtain a second feature map of the current level and pass the second feature map of the current level to a next deconvolution layer connected with the current deconvolution layer.

9. An electronic device, comprising: including: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, a memory having executable code stored thereon, which, when executed by the processor of the electronic device, causes the processor to perform the method of any one of claims 1-7.

Images