Image processing method, device, equipment and storage medium

Abstract

The present application provides an image processing method, apparatus, device and storage medium, which obtain an image to be processed; perform feature extraction on the image to be processed n times to obtain a pixel feature recognition result; each feature extraction includes p data nodes, where p is an integer greater than 1; the data on one of the data nodes is determined by processing the data of each data node before the data node in the current feature extraction process using a set feature extraction method and the input data of the current feature extraction; the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in the feature extraction process, and the feature extraction methods used in each feature extraction process are not completely the same; and the image to be processed is processed according to the pixel feature recognition result.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of deep learning technology, image recognition has been widely promoted and applied. One of the important contents of image recognition is to obtain image feature recognition results by performing multiple feature extraction processes on the image, and then complete the recognition process of the image content. Among them, the diversity and rationality of feature extraction processing methods are important factors that determine the quality of image recognition results.

[0003] In the prior art, when performing feature extraction on an image, a neural network search method is used to search for the best neural network unit, and then the searched neural network units are stacked to form a whole neural network to perform feature extraction on the image. That is to say, a neural network search method is used to first determine an optimal feature extraction method, and the determined feature extraction method is used to perform feature extraction on the image for multiple times to obtain image feature recognition results, and then the obtained image feature recognition results are used to complete the recognition of the image content.

[0004] However, using the determined feature extraction method to perform multiple feature extraction processes on the image will cause the obtained image feature recognition results to have problems of poor accuracy and large granularity. When using the image feature recognition results to perform image content recognition, there is a defect of low image recognition accuracy. Summary of the invention

[0005] The present application provides an image processing method, apparatus, device and storage medium to solve the problem of low image content recognition accuracy in existing image processing methods.

[0006] In a first aspect, the present application provides an image processing method, comprising:

[0007] Get the image to be processed;

[0008] Performing feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed; wherein the input data of each feature extraction is determined based on the output data of the previous k feature extractions; the input data of the mth feature extraction includes the image to be processed; wherein m is an integer greater than or equal to 1 and less than or equal to k, k is an integer greater than or equal to 1 and less than n, and n is an integer greater than 1;

[0009] Each feature extraction process includes p data nodes, where p is an integer greater than 1;

[0010] The data on a data node in each feature extraction process is determined based on the data of each data node before the data node in the current feature extraction process, which is processed by using the set feature extraction method and the input data of the current feature extraction; wherein the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in the feature extraction process, and the feature extraction methods used in each feature extraction process are not completely the same; the output data of each feature extraction process is the data on the last data node in the current feature extraction process;

[0011] The image to be processed is processed according to the pixel feature recognition result of the image to be processed.

[0012] In combination with one or more embodiments of the present application, the image processing method is performed by a neural network, and each feature extraction process is performed by a layer of basic units in the neural network.

[0013] In combination with one or more embodiments of the present application, the feature extraction method used in each feature extraction process is determined according to the following steps:

[0014] Repeat the following steps until the feature extraction method used by the data on each data node in the feature extraction process is determined for any subsequent data node in the feature extraction process:

[0015] Select any data node in the feature extraction and any data node after the data node to form a target data node pair, wherein there are multiple candidate feature extraction methods for processing the data of the previous data node in the target data node pair to the next data node in the target data node pair;

[0016] Using the weight value of each candidate feature extraction method in the process from the previous data node to the next data node, a first connection vector is formed, wherein the weight value of each candidate feature extraction method is an arbitrary value;

[0017] Normalizing the first connection vector to obtain a first eigenvector, wherein only one element of the first eigenvector is 1, and the other elements are all 0;

[0018] The candidate feature extraction method corresponding to the determined element 1 is the feature extraction method adopted from the data of the previous data node to the next data node.

[0019] In combination with one or more embodiments of the present application, before determining the feature extraction method used in the latter feature extraction process of two adjacent feature extraction processes, the method further includes:

[0020] Using graph convolution to correct each first feature vector in the subsequent feature extraction process according to the feature extraction method used in the previous feature extraction process of the two adjacent feature extraction processes;

[0021] The feature extraction method used in the subsequent feature extraction process is determined according to the corrected first feature vectors.

[0022] In combination with one or more embodiments of the present application, each first feature vector in the subsequent feature extraction process is corrected according to the feature extraction method used in the previous feature extraction process of the two adjacent feature extraction processes by using a graph convolution method, including:

[0023] Determine a connection matrix between a first matrix and a second matrix, wherein the first matrix is ​​a matrix formed by the feature extraction method used in the previous feature extraction process, and the second matrix is ​​a matrix formed according to each first feature vector in the next feature extraction process;

[0024] Determine a correlation value between the connection matrix and the first matrix, and add the correlation value to the second matrix to obtain a third matrix;

[0025] The corrected first eigenvectors are determined according to the third matrix.

[0026] In combination with one or more embodiments of the present application, the image to be processed is processed according to the pixel feature recognition result of the image to be processed, including:

[0027] According to the pixel feature recognition result of the image to be processed, performing feature extraction of at least one scale on the pixel feature recognition result to obtain a feature extraction result of at least one scale of the image to be processed;

[0028] The image to be processed is processed according to a feature extraction result of at least one scale of the image to be processed.

[0029] In combination with one or more embodiments of the present application, the following steps are used to determine the feature extraction method for each scale:

[0030] Using weight values ​​of various candidate feature extraction methods corresponding to a feature extraction method of a scale to form a second connection vector, wherein the weight value of each candidate feature extraction method is an arbitrary value;

[0031] Normalizing the second connection vector to obtain a second eigenvector, wherein only one element of the second eigenvector is 1, and the other elements are all 0;

[0032] It is determined that the candidate feature extraction method corresponding to element 1 is the feature extraction method of the scale.

[0033] In combination with one or more embodiments of the present application, the input data for each feature extraction is determined according to the output data of the previous k feature extractions using the following steps:

[0034] The output data of each feature extraction in k feature extractions before one feature extraction is processed by a set processing method to obtain k processing results;

[0035] The input data for this feature extraction is determined according to the k processing results.

[0036] In a second aspect, the present application provides an image processing device, comprising:

[0037] An acquisition unit, used for acquiring an image to be processed;

[0038] A first processing unit is used to perform feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed; wherein the input data of each feature extraction is determined according to the output data of the previous k feature extractions; the input data of the mth feature extraction includes the image to be processed; wherein m is an integer greater than or equal to 1 and less than or equal to k, k is an integer greater than or equal to 1 and less than n, and n is an integer greater than 1;

[0039] Each feature extraction process includes p data nodes, where p is an integer greater than 1;

[0040] The data on a data node in each feature extraction process is determined based on the data of each data node before the data node in the current feature extraction process, which is processed by using the set feature extraction method and the input data of the current feature extraction; wherein the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in the feature extraction process, and the feature extraction methods used in each feature extraction process are not completely the same; the output data of each feature extraction process is the data on the last data node in the current feature extraction process;

[0041] The second processing unit is used to process the image to be processed according to the pixel feature recognition result of the image to be processed.

[0042] In combination with one or more embodiments of the present application, the first processing unit includes a first processing sub-unit, which is used to:

[0043] Repeat the following steps until the feature extraction method used by the data on each data node in the feature extraction process is determined for any subsequent data node in the feature extraction process:

[0044] Select any data node in the feature extraction and any data node after the data node to form a target data node pair, wherein there are multiple candidate feature extraction methods for processing the data of the previous data node in the target data node pair to the next data node in the target data node pair;

[0045] Using the weight value of each candidate feature extraction method in the process from the previous data node to the next data node, a first connection vector is formed, wherein the weight value of each candidate feature extraction method is an arbitrary value;

[0046] Normalizing the first connection vector to obtain a first eigenvector, wherein only one element of the first eigenvector is 1, and the other elements are all 0;

[0047] The candidate feature extraction method corresponding to the determined element 1 is the feature extraction method adopted from the data of the previous data node to the next data node.

[0048] In combination with one or more embodiments of the present application, the first processing unit further includes:

[0049] The second processing subunit is used to modify each first feature vector in the next feature extraction process by using a graph convolution method according to the feature extraction method used in the previous feature extraction process of two adjacent feature extraction processes;

[0050] The third processing subunit is used to determine the feature extraction method used in the next feature extraction process according to the corrected first feature vectors.

[0051] In combination with one or more embodiments of the present application, the second processing subunit includes:

[0052] A first processing module, used to determine a connection matrix between a first matrix and a second matrix, wherein the first matrix is ​​a matrix formed by the feature extraction method used in the previous feature extraction process, and the second matrix is ​​a matrix formed according to each first feature vector in the next feature extraction process;

[0053] A second processing module, used for determining a correlation value between the connection matrix and the first matrix, and adding the correlation value to the second matrix to obtain a third matrix;

[0054] The third processing module is used to determine the corrected first eigenvectors according to the third matrix.

[0055] In combination with one or more embodiments of the present application, the second processing unit includes:

[0056] a fourth processing subunit, configured to perform feature extraction of at least one scale on the pixel feature recognition result of the image to be processed according to the pixel feature recognition result of the image to be processed, so as to obtain a feature extraction result of at least one scale of the image to be processed;

[0057] The fifth processing subunit is used to process the image to be processed according to the feature extraction result of at least one scale of the image to be processed.

[0058] In combination with one or more embodiments of the present application, the fourth processing subunit includes:

[0059] A fourth processing module, configured to form a second connection vector by using weight values ​​of various candidate feature extraction methods corresponding to a feature extraction method of a scale, wherein the weight value of each candidate feature extraction method is an arbitrary value;

[0060] a fifth processing module, configured to perform normalization processing on the second connection vector to obtain a second eigenvector, wherein only one element of the second eigenvector is 1, and the other elements are all 0;

[0061] The sixth processing module is used to determine that the candidate feature extraction method corresponding to element 1 is the feature extraction method of the scale.

[0062] In combination with one or more embodiments of the present application, the first processing unit further includes:

[0063] The sixth processing subunit is used to process the output data of each feature extraction in k feature extractions before the first feature extraction by a set processing method to obtain k processing results;

[0064] The seventh processing subunit is used to determine the input data for this feature extraction according to the k processing results.

[0065] In a third aspect, the present application provides an image processing device, including: a processor, a memory, and a computer program;

[0066] The computer program is stored in the memory and is configured to be executed by the processor to implement any of the above methods.

[0067] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement any of the above methods.

[0068] The present application provides an image processing method, apparatus, device and storage medium, which obtain an image to be processed; perform feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed; the data on a data node in each feature extraction process is determined by processing the data of each data node before the data node in this feature extraction process using a set feature extraction method and the input data of this feature extraction; wherein the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in this feature extraction process, and the feature extraction methods used in each feature extraction process are not completely the same; the output data of each feature extraction process is the data on the last data node in this feature extraction process; and content recognition processing is performed on the image to be processed according to the pixel feature recognition result of the image to be processed. Since the feature extraction methods used in each feature extraction process are not exactly the same, that is to say, searching out multiple neural network units with different structures, performing multiple feature extractions is equivalent to using multiple neural network units with different structures to form a neural network to perform feature extraction on the image to be processed, thereby ensuring the diversity of feature extraction processing when performing pixel feature recognition on the image. In addition, since the feature extraction methods in multiple feature extraction processes are independent of each other, the independence of various feature extraction methods is guaranteed, which is conducive to achieving more accurate and fine-grained pixel feature recognition results, and is conducive to improving the recognition accuracy of the image recognition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0070] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0071] Figure 2 A flowchart of another image processing method provided in an embodiment of the present application;

[0072] Figure 3 A schematic diagram of the structure of a basic unit provided in an embodiment of the present application;

[0073] Figure 4 A schematic diagram of the structure of a basic unit after the processing layer between data nodes provided in an embodiment of the present application is determined;

[0074] Figure 5 A schematic diagram of the connection between the feature unit provided in this embodiment and the last layer of basic units;

[0075] Figure 6A schematic diagram of the connection structure between the feature unit after the processing layer is determined and the last layer of basic units provided in an embodiment of the present application;

[0076] Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0077] Figure 8 A schematic diagram of the structure of another image processing device provided in an embodiment of the present application;

[0078] Fig. 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0079] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0080] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0081] Explanation of terms involved in this application:

[0082] Topology: The way in which nodes in a network are connected to each other.

[0083] Softmax function: also known as the normalized exponential function, can normalize the logarithm of the gradient of a finite discrete probability distribution.

[0084] Semantic segmentation: also known as image segmentation, is a technique and process for dividing an image into several specific regions with unique properties and proposing objects of interest. Its essence is the process of assigning labels to each pixel in the image.

[0085] Image recognition: technology that uses computers to process, analyze and understand images to identify targets and objects in various different patterns.

[0086] Feature extraction: Use computers to extract image information and determine whether the points that make up the image belong to an image feature. The result of feature extraction is to divide the points on the image into different subsets, which can be isolated points, continuous curves or continuous areas.

[0087] Graph Convolutional Neural Network: GCN, an algorithm for deep learning on graph data.

[0088] Image features: including image color features, texture features, shape features, and spatial relationship features.

[0089] The image processing method, apparatus, device and storage medium provided in the present application are intended to perform multiple feature extractions on the image to be processed, and the feature extraction methods used in the multiple feature extraction processes are not exactly the same, so as to achieve more accurate and fine-grained pixel feature recognition results to improve the recognition accuracy of the image recognition process.

[0090] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, including:

[0091] Step 101: Obtain an image to be processed.

[0092] In this embodiment, specifically, the execution subject of this embodiment is a terminal, or a controller, or other devices or equipment that can execute this embodiment. This embodiment is described with the execution subject being a terminal, and application software can be set in the terminal, and then the terminal controls the application software to execute the method provided by this embodiment.

[0093] Obtain an image to be processed. The image to be processed may contain different object regions. For example, the image to be processed contains two object regions, a person and a motorcycle. It is necessary to use image recognition technology to segment different object regions from the image to be processed, mark the boundaries of the object regions, and identify the contents of the object regions. The source of the image to be processed can be an image taken by a camera, or a network image, a screenshot image, or an electronic scanned image.

[0094] Step 102: perform feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed, wherein the input data of each feature extraction is determined based on the output data of the previous k feature extractions; the input data of the mth feature extraction includes the image to be processed; wherein m is an integer greater than or equal to 1 and less than or equal to k, k is an integer greater than or equal to 1 and less than n, and n is an integer greater than 1.

[0095] Each feature extraction process includes p data nodes; wherein p is an integer greater than 1; the data on a data node in each feature extraction process is determined by processing the data of each data node before the data node in this feature extraction process using a set feature extraction method and the input data of this feature extraction; wherein the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in this feature extraction process, and the feature extraction methods used in each feature extraction process are not exactly the same; the output data of each feature extraction process is the data on the last data node in this feature extraction process.

[0096] In this embodiment, specifically, feature extraction is performed on the image to be processed n times, n is an integer greater than 1, and the input data of each feature extraction is determined based on the output data of the previous feature extraction or multiple previous feature extractions, that is, the input data of each feature extraction is determined based on the output data of the previous k feature extractions, k is an integer greater than or equal to 1 and less than n, wherein the input data of the mth feature extraction includes the image to be processed, and m is an integer greater than or equal to 1 and less than or equal to k. When k=1, m=1, that is, when the input data of each feature extraction is determined based on the output data of the previous feature extraction, the input data of the first feature extraction is the image to be processed; when k>1, m=1, ..., k, that is, when the input data of each feature extraction is determined based on the output data of multiple previous feature extractions, there are more than one input data of feature extraction including the image to be processed. For example, k=3, then m=1, 2, 3, and the input data of the 1st, 2nd and 3rd feature extractions are all determined based on the image to be processed. When m=4, the input data of the mth feature extraction is determined based on the output data of the 1st, 2nd and 3rd feature extractions.

[0097] Each feature extraction process includes p data nodes, where p is an integer greater than 1. The data of one data node is determined by the data of each data node before the data node in the feature extraction process obtained by using the set feature extraction method and the input data of this feature extraction. For the first data node, the data of the first data node is the input data of this feature extraction. The data of the first data node is processed by the set feature extraction method, and the data after each feature extraction is passed to each data node after the first data node; for a certain data node in the middle, the data of the data node is determined by the data of each data node before the data node obtained by using the set feature extraction method and the input data of this feature extraction. The data of the data node is processed by the set feature extraction method, and the data after each feature extraction method is passed to each data node after the data node; for the last data node, the data of the data node is determined by the data of each data node before the data node obtained by using the set feature extraction method and the input data of this feature extraction. The data of the last data node is the output data of this feature extraction and is passed to the first data node of the 1st to kth feature extraction after this feature extraction. During each feature extraction process, the set feature extraction methods used for the data of other data nodes except the last data node are independent of each other. Among them, the number of times the data of a data node except the last data node is processed for feature extraction using the set feature extraction method is determined by the number of data nodes after the data node.

[0098] Exemplarily, taking p=4 as an example, a feature extraction process includes 4 data nodes. For the first data node, the data of the first data node is the input data of this feature extraction. The data of the first data node is subjected to three feature extraction processes respectively using the set feature extraction method, and the data obtained by the three feature extraction processes are respectively passed to the second, third, and fourth data nodes, and the feature extraction methods used in the three feature extraction processes are independent of each other; for the second data node, the data of the second data node is determined by the data obtained by processing the data of the first data node using the set feature extraction method and the input data of this feature extraction. The data of the second data node is subjected to two feature extraction processes respectively using the set feature extraction method, and the data obtained by the two feature extraction processes are respectively passed to the third and fourth data nodes; for the fourth data node, the data of the fourth data node is determined by the data obtained by processing the data of the first, second, and third data nodes using the set feature extraction method and the input data of this feature extraction, and the data of the fourth data node is the output data of this feature extraction process.

[0099] Step 103: Process the image to be processed according to the pixel feature recognition result of the image to be processed.

[0100] In this embodiment, specifically, the image to be processed is processed according to the obtained pixel feature recognition result, which can be specifically content recognition processing. The content recognition processing is to determine the object category of each pixel of the image to be processed, determine the pixels belonging to the same object category as the same pixel feature recognition result, divide the pixels belonging to the same pixel feature recognition result into the same object area, and assign the same number to the pixels in the same object area. By dividing the image to be processed into a number of non-intersecting object areas and marking different object areas, the content recognition process of the image to be processed is realized.

[0101] This embodiment obtains an image to be processed; performs feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed; the data on a data node in each feature extraction process is determined by processing the data of each data node before the data node in this feature extraction process using a set feature extraction method and the input data of this feature extraction; wherein the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in this feature extraction process, and the feature extraction methods used in each feature extraction process are not completely the same; the output data of each feature extraction process is the data on the last data node in this feature extraction process; and the image to be processed is processed according to the pixel feature recognition result of the image to be processed. Since the feature extraction methods used in each feature extraction process are not exactly the same, that is to say, searching out multiple neural network units with different structures, performing multiple feature extractions is equivalent to using multiple neural network units with different structures to form a neural network to perform feature extraction on the image to be processed, thereby ensuring the diversity of feature extraction processing when performing pixel feature recognition on the image. In addition, since the feature extraction methods in multiple feature extraction processes are independent of each other, the independence of various feature extraction methods is guaranteed, which is conducive to achieving more accurate and fine-grained pixel feature recognition results, and is conducive to improving the recognition accuracy of the image recognition process.

[0102] Figure 2 A flowchart of another image processing method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method includes:

[0103] Step 201: Obtain an image to be processed.

[0104] In this embodiment, specifically, this step can be referred to Figure 1Step 101 in is not described in detail.

[0105] Step 202: Perform feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed.

[0106] The image processing method is performed by a neural network, and each feature extraction process is performed by a layer of basic units in the neural network. The neural network is composed of at least n layers of basic units, and each layer of basic units includes p data nodes, where n is an integer greater than 1 and p is an integer greater than 1.

[0107] Using a neural network to extract features of an image to be processed n times includes: inputting the image to be processed into the first layer of basic units of the neural network; using the first layer of basic units to extract features of the image to be processed to obtain a first processing result; repeating the following steps until the feature extraction process is completed using each layer of basic units of the neural network to obtain a pixel feature recognition result of the image to be processed: inputting the first processing result of the previous k layers of basic units into the adjacent next layer of basic units, and using the next layer of basic units to extract features of the first processing result of the previous k layers of basic units to obtain a second processing result. Wherein, the input data of the mth layer of basic units includes the image to be processed, wherein m is an integer greater than or equal to 1 and less than or equal to k, and k is an integer greater than or equal to 1 and less than n.

[0108] The image to be processed is input into the first layer of basic units of the neural network for feature extraction processing, and then the first processing results of the previous k layers of basic units are sequentially input into the adjacent next layer of basic units until the feature extraction processing using each layer of basic units of the neural network is completed to obtain the pixel feature recognition results of the image to be processed. The feature extraction processing operation performed by each layer of basic units constitutes the feature extraction processing operation performed by the entire neural network on the image to be processed. The feature extraction processing method adopted by each layer of basic units includes the feature extraction method adopted by the data of all data nodes in the basic unit. When using each layer of basic units for feature extraction processing, the feature extraction processing methods adopted by each basic unit are independent of each other. The diversity of the feature extraction processing methods performed by each basic unit on the image to be processed is conducive to improving the recognition accuracy of the image recognition process.

[0109] The feature extraction method used in each feature extraction process is determined according to the following steps:

[0110] Repeat the following steps until the feature extraction method used by the data on each data node in a feature extraction to any subsequent data node in this feature extraction process is determined: select any data node in this feature extraction and any data node after the data node to form a target data node pair, wherein the processing of the data of the previous data node in the target data node pair to the next data node in the target data node pair has multiple candidate feature extraction methods; use the weight value of each candidate feature extraction method for the processing from the previous data node to the next data node to form a first connection vector, wherein the weight value of each candidate feature extraction method is an arbitrary value; normalize the first connection vector to obtain a first feature vector, wherein only one element in the first feature vector is 1, and the rest of the elements are 0; determine that the candidate feature extraction method corresponding to element 1 is the feature extraction method used by the data of the previous data node to the next data node.

[0111] When a basic unit of a neural network is used for feature extraction, the data of the previous data node is processed in turn using the set feature extraction method, and the processed data is passed to each data node after the data node, until the feature extraction process is completed for the data of each data node of the basic unit, and the pixel feature recognition result of the basic unit is obtained. The feature extraction process used for the data of each data node constitutes the feature extraction process of the basic unit, and the feature extraction process operation performed on the data of each data node constitutes the feature extraction process operation of the basic unit.

[0112] Different data nodes are marked with preset node numbers, and the data nodes are sorted according to the order of node numbers from small to large (for node numbers that are numbers) or from front to back (for node numbers that are other non-numeric numbers, such as English letters, etc.). Figure 3 A schematic diagram of the structure of the basic unit provided in the embodiment of the present application, such as Figure 3 As shown, the basic unit includes 4 data nodes, and the node numbers of the 4 data nodes are 0, 1, 2, and 3 respectively. The data nodes are sorted from front to back according to the order of the node numbers from small to large. The data node with the node number 0 is the first data node, and the data node with the node number 3 is the last data node.

[0113] In the node sequence number list, the node sequence number of a data node is located in front of the node sequence numbers of each data node after the data node. For example, in the node sequence number list, node sequence number 0 is located in front of node sequence number 1, and data node No. 0 is located in front of data node No. 1, such as Figure 3As shown, for data node No. 0, each data node after the data node includes data nodes No. 1, 2, and 3; for data node No. 1, each data node after the data node includes data nodes No. 2 and 3.

[0114] When any layer of basic units of a neural network is used for feature extraction processing, any data node in the feature extraction and any data node after the data node are selected to form a target data node pair. In the target data node pair, the data of the previous data node is processed by a specific feature extraction method, and the processed data is passed to the next data node. The data of the previous data node of the target data node pair corresponds to at least one candidate feature extraction method, and the candidate feature extraction methods include convolution processing, pooling processing, direct permutation processing, etc. The feature extraction operation between data nodes is implemented by the processing layer between data nodes. Correspondingly, the processing layer includes a convolution layer, a pooling layer, a direct permutation layer, etc. The processing layer is used to process the data of the previous data node and pass the processing result to the next data node. Exemplarily, data nodes No. 1 and No. 2 constitute a target data node pair, and the data of data node No. 1 corresponds to three candidate feature extraction methods, which are convolution processing, pooling processing, and direct replacement processing. Convolution processing is to perform convolution processing on the data of data node No. 1, and input the processing result to data node No. 2; direct replacement processing is to do no processing on the data of data node No. 1, and directly replace the output result with the input of data node No. 2.

[0115] In the target data node pair, each candidate feature extraction method corresponding to the data of the previous data node has a corresponding weight value, wherein the weight value can be an arbitrary value. The weight values ​​of all candidate feature extraction methods constitute the first connection vector. The first connection vector is normalized to obtain a one-hot vector, that is, a first feature vector in which only one element is 1 and the rest of the elements are 0. The method for normalizing the first connection vector can be implemented using the Softmax function. In the obtained first feature vector, the feature extraction method corresponding to element 1 is the feature extraction method corresponding to the data of the previous data node. After the data of the previous data node is processed using the feature extraction method corresponding to element 1, the processed data is passed to the next data node.

[0116] For example, Figure 3As shown, the first and second data nodes constitute a target data node pair, and the data of the first data node corresponds to three candidate feature extraction methods, which are convolution processing, pooling processing, and direct replacement processing, that is, there are three candidate processing layers between the first data node and the second data node, which are convolution layer, pooling layer, and direct replacement layer. The weight values ​​of the three candidate feature extraction methods are a, b, and c, respectively. The weight values ​​a, b, and c constitute the first connection vector L = (a, b, c). The first connection vector L = (a, b, c) is normalized using the Softmax function to obtain the first feature vector Z = (0, 1, 0), where the feature extraction method corresponding to element 1 is the feature extraction method used by the data of the first data node. Therefore, the processing layer between the first data node and the second data node is the pooling layer, that is, the data on the first data node is used as one of the data on the second data node after pooling. After determining the feature extraction method between any two data nodes in the data node from front to back direction, that is, after determining the feature extraction method used between data nodes No. 0 and 1, data nodes No. 0 and 2, data nodes No. 0 and 3, data nodes No. 1 and 2, data nodes No. 1 and 3, and data nodes No. 2 and 3, the feature extraction method used for feature extraction processing using the basic unit is obtained. Determining the feature extraction method used by a basic unit for feature extraction processing is the process of determining the internal structure of a basic unit, that is, the process of searching for neural network units.

[0117] Figure 4 A schematic diagram of the structure of the basic unit after the processing layer between the data nodes provided in the embodiment of the present application is determined, such as Figure 4 As shown, after the data nodes are determined by the feature extraction method used between any two data nodes in the front and back connection directions, the processing layer between the data nodes is one of all processing layer types.

[0118] Before determining the feature extraction method used in the latter feature extraction process of two adjacent feature extraction processes, the method further includes:

[0119] A graph convolution method is used to correct each first feature vector in a subsequent feature extraction process according to the feature extraction method used in the previous feature extraction process of two adjacent feature extraction processes; and the feature extraction method used in the subsequent feature extraction process is determined based on the corrected first feature vectors.

[0120] Specifically, determine a connection matrix between the first matrix and the second matrix, wherein the first matrix is ​​a matrix formed by the feature extraction method used in the previous feature extraction process, and the second matrix is ​​a matrix formed according to each first eigenvector in the next feature extraction process; determine the correlation value between the connection matrix and the first matrix, and add the correlation value to the second matrix to obtain a third matrix; determine each corrected first eigenvector based on the third matrix.

[0121] The basic units are numbered using preset unit numbers, and the unit numbers of all basic units constitute a unit number list. The earlier the unit number is in the unit number list, the higher the level of the corresponding basic unit. For example, in the unit number list (0, 1, 2, ..., 10), the basic unit with unit number 0 is the first-layer basic unit of the neural network, and the basic unit with unit number 10 is the last-layer basic unit of the neural network.

[0122] This method uses a graph convolutional network as a communication mechanism between basic units. In two adjacent feature extraction processes, the feature extraction method used in the previous feature extraction process is used to make decisions for the feature extraction method in the next feature extraction process, which can effectively improve the recognition accuracy and recognition efficiency of image recognition. Determine the association value between the first matrix formed by the feature extraction method used in the previous feature extraction process and the second matrix formed by each first feature vector in the next feature extraction process, and add the calculated association value to the second matrix of the next feature extraction process to obtain a third matrix, and then correct each first feature vector in the next feature extraction process according to the third matrix, and the obtained first feature vectors constitute the corrected feature extraction method of the next feature extraction process. Repeat the above steps until the corrected feature extraction methods of all basic units except the first-layer basic unit are determined. Because there are no other basic units before the first-layer basic unit, it is impossible to use the feature extraction methods of other basic units to make decisions for its feature extraction process. Therefore, the feature extraction method of the feature extraction processing performed by the first-layer basic unit is determined by the first feature vector of each data node of the basic unit. In any two adjacent basic units of the neural network, the method for determining the upper basic unit and the lower basic unit includes: in the unit number list, the basic unit with the earlier unit number is the upper basic unit, and the basic unit with the later unit number is the lower basic unit.

[0123] In two adjacent feature extraction processes, a method for determining the correlation value between a first matrix formed by a feature extraction method used in a previous feature extraction process and a second matrix formed by first feature vectors in a subsequent feature extraction process includes: using matrix α n-1 Represents the first matrix, using matrix α nRepresents the second matrix, and uses the matrix Adj to represent α n-1 With α n The connection matrix of Represents α n-1 The transposed matrix of the connection matrix can be used to describe the connection relationship between the two matrices; the graph convolutional neural network GCN is used to calculate the association value Δα=GCN(Adj,α n-1 ,W), where W is a preset parameter. The matrix α can also be n-1 Input the fully connected layer, perform linear feature processing, and input the processed matrix into the graph convolutional neural network GCN to obtain the association value. After calculating the association value Δα, add the association value Δα to the first matrix α n , and obtain the third matrix. According to the third matrix, determine the first eigenvector of each data node after correction in the next feature extraction process. Repeat the above steps until the first eigenvector of each data node after correction of all basic units except the first-layer basic unit is calculated, and determine the corrected privilege extraction method of all other feature extraction processes except the first feature extraction process.

[0124] According to the first eigenvector of each data node of the first-layer basic unit, the processing layer between each data node of the first-layer basic unit is determined, that is, the topological structure of the first-layer basic unit is determined, and then the first-layer basic unit can be searched out according to the determined topological structure. According to the corrected first eigenvector of each data node of the basic units of other layers except the first-layer basic unit, the processing layer between each data node of the basic units of other layers is determined, that is, the topological structure of the basic units of other layers is determined, and then the basic units of other layers can be searched out according to the determined topological structure, and then the whole neural network can be searched out by the search method.

[0125] In each feature extraction process, the feature extraction processing methods used for the data of the previous data node of any two target data nodes are independent of each other, so the feature extraction methods used in each feature extraction process are independent and diverse, and the multiple feature extraction processes performed on the processed image using the neural network are diverse. When the basic units (i.e., neural network units) obtained by the above search method are used to form a neural network for image recognition processing, the exponential reduction of the feature extraction method can be effectively avoided, which is conducive to ensuring the diversity and extensiveness of the feature extraction method of the neural network. At the same time, the feature extraction method of the previous feature extraction process is used to provide a decision for the feature extraction method of the next feature extraction process. By using the graph convolutional network as a communication mechanism, a more optimized feature extraction method is decided for the next feature extraction process, which can effectively improve the recognition accuracy and efficiency of image recognition.

[0126] Step 203: perform feature extraction of at least one scale on the pixel feature recognition result according to the pixel feature recognition result of the image to be processed, to obtain a feature extraction result of at least one scale of the image to be processed.

[0127] In this embodiment, specifically, the feature unit of the neural network is used to perform feature extraction of at least one scale on the pixel feature recognition result to obtain feature extraction results of different scales of the image to be processed. The data obtained by the last feature extraction process is the pixel feature recognition result of the image to be processed, the feature unit includes at least one feature data node, the pixel feature recognition result is input into each feature data node of the feature unit respectively, and the pixel feature recognition result of each feature data node is subjected to feature extraction of at least one scale to obtain a feature extraction result of at least one scale corresponding to each feature data node.

[0128] The method for determining feature extraction of each scale includes: using the weight values ​​of various candidate feature extraction methods corresponding to the feature extraction method of one scale to form a second connection vector, wherein the weight value of each candidate feature extraction method is an arbitrary value; normalizing the second connection vector to obtain a second feature vector, wherein only one element in the second feature vector is 1, and the remaining elements are 0; and determining that the candidate feature extraction method corresponding to element 1 is the feature extraction method of the scale.

[0129] The basic unit performs feature extraction processing based on a single pixel on the image to be processed. After using each layer of the basic unit of the neural network to perform feature extraction processing on the image to be processed, the pixel feature recognition result of the image to be processed is obtained. The feature unit is used to perform feature extraction based on multiple pixels on the pixel feature recognition result, that is, it is used to perform feature extraction of at least one scale on the pixel feature recognition result to obtain a feature extraction result of at least one scale of the image to be processed. The feature unit includes at least one feature data node, each feature data node performs a feature extraction processing on the pixel feature recognition result, and the feature extraction methods used by each feature data node are independent of each other. Different feature data nodes can perform feature extraction processing of different scales on the pixel feature recognition result, wherein feature extraction of different scales refers to global feature extraction, or various local feature extractions.

[0130] In this embodiment, specifically, the feature unit is composed of at least one feature data node, and each feature data node has at least one optional feature extraction method for the pixel feature recognition result, wherein the optional feature extraction methods include convolution processing, pooling processing, direct replacement processing, etc. Specifically, there are multiple processing layers between each feature data node and the bottom layer of basic units, and the processing layers include convolution layers, pooling layers, direct replacement layers, etc. Exemplarily, Figure 5The connection diagram of the feature unit 502 and the bottom layer basic unit 501 provided in this embodiment is as follows: Figure 5 As shown, the feature unit 502 is composed of 5 feature data nodes. There are 12 candidate processing layers between each feature data node and the lowest basic unit 501. The 12 candidate processing layers include: depthwise separable convolutional layers with widths of 6, 10, 12, and 18, global average pooling layers with sizes of 2, 3, 5, 6, and 10, 1x1 convolutional layers, and direct permutation layers.

[0131] Select any feature data node as the target feature data node, and use the weight values ​​of all candidate feature extraction methods of the target feature data node for the pixel feature recognition result to form a second connection vector, wherein the weight value of each candidate feature extraction method is an arbitrary value. The second connection vector is normalized to obtain a one-hot vector, that is, a second feature vector with only one element being 1 and the rest being 0. The method for normalizing the second connection vector can be implemented using a Softmax function. In the obtained second feature vector, the candidate feature extraction method corresponding to element 1 is the feature extraction method performed by the target feature data node on the pixel feature recognition result. After determining the feature extraction method performed by all feature data nodes on the pixel feature recognition result, the type of processing layer between all feature data nodes and the last basic unit layer is determined, that is, the topological structure of the feature unit is determined. According to the determined topological structure, the corresponding feature unit can be searched for to form the entire neural network.

[0132] Determine the feature extraction method of each feature data node for the pixel feature recognition result, that is, determine the processing layer between each feature data node and the last layer of basic units, Figure 6 A schematic diagram of the connection structure between the feature unit 602 after the processing layer is determined and the last layer of the basic unit 601 provided in the embodiment of the present application, as shown in FIG. Figure 6 As shown, each feature data node constituting the feature unit 602 and the processing layer type of the last layer basic unit 601 are determined, that is, each feature data node constituting the feature unit 602 has a determined feature extraction method for the pixel feature recognition result.

[0133] Step 204: Process the image to be processed according to the feature extraction result of at least one scale of the image to be processed.

[0134] Exemplarily, processing the image to be processed according to the feature extraction result of at least one scale of the image to be processed may include performing pixel feature classification on the image to be processed according to the feature extraction result of at least one scale of the image to be processed. In the pixel feature classification result, pixels belonging to the same object category are determined to be in the same pixel feature classification, and then the pixels belonging to the same pixel feature classification are divided into the same object area, and the pixels in the same object area are assigned the same number, thereby completing content recognition of the image to be processed.

[0135] This embodiment obtains an image to be processed; obtains the image to be processed; performs feature extraction n times on the image to be processed to obtain a pixel feature recognition result of the image to be processed, wherein the feature extraction methods used in each feature extraction process are not completely the same; based on the pixel feature recognition result of the image to be processed, performs feature extraction of at least one scale on the pixel feature recognition result to obtain a feature extraction result of at least one scale of the image to be processed; and performs content recognition processing on the image to be processed based on the feature extraction result of at least one scale of the image to be processed. In each feature extraction process, the feature extraction processing methods used by any two data nodes for the data of the previous data nodes are independent, which ensures the independence and diversity of the feature extraction methods for feature extraction using the basic unit, and the obtained pixel feature recognition results are highly accurate and have great reference value; using the graph convolutional network as a communication mechanism, in two adjacent feature extractions using the basic unit, the feature extraction method of the previous feature extraction is used to provide a decision for the feature extraction method of the next feature extraction, and by using the graph convolutional network to decide a more optimized feature extraction method, the recognition accuracy of the pixel feature recognition results can be effectively guaranteed; the feature extraction method used by each feature data node of the feature unit that constitutes the neural network is independent, which ensures the diversity of feature extraction methods when the feature unit is used to extract features of at least one scale of the pixel feature recognition results, which is conducive to improving the accuracy of the image recognition results; since the feature extraction methods used in each feature extraction process are not exactly the same, this can effectively avoid the exponential reduction of the feature extraction method, which is conducive to ensuring the diversity and extensiveness of the processing methods of the neural network, and can effectively ensure the image recognition performance of the neural network, and can effectively improve the image recognition accuracy.

[0136] Figure 7 A schematic diagram of the structure of an image recognition processing device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device comprises:

[0137] An acquisition unit 1 is used to acquire an image to be processed;

[0138] The first processing unit 2 is used to perform feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed; wherein the input data of each feature extraction is determined according to the output data of the previous k feature extractions; the input data of the mth feature extraction includes the image to be processed; wherein m is an integer greater than or equal to 1 and less than or equal to k, k is an integer greater than or equal to 1 and less than n, and n is an integer greater than 1;

[0139] Each feature extraction process includes p data nodes, where p is an integer greater than 1;

[0140] The data on a data node in each feature extraction process is determined based on the data of each data node before the data node in the current feature extraction process, which is processed by using the set feature extraction method and the input data of the current feature extraction; wherein the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in the feature extraction process, and the feature extraction methods used in each feature extraction process are not completely the same; the output data of each feature extraction process is the data on the last data node in the current feature extraction process;

[0141] The second processing unit 3 is used to process the image to be processed according to the pixel feature recognition result of the image to be processed.

[0142] This embodiment obtains the image to be processed; performs n feature extractions on the image to be processed to obtain the pixel feature recognition result of the image to be processed; the data on a data node in each feature extraction process is determined by the data of each data node before the data node in this feature extraction process respectively processed by the set feature extraction method and the input data of this feature extraction; wherein, the feature extraction method adopted in a feature extraction process includes the feature extraction method adopted by the data on each data node in this feature extraction process, and the feature extraction methods adopted in each feature extraction process are not completely the same; the output data of each feature extraction process is the data on the last data node in this feature extraction process; according to the pixel feature recognition result of the image to be processed, the image to be processed is processed. Since the feature extraction methods adopted in each feature extraction process are not completely the same, this ensures the diversity of feature extraction processing when performing pixel feature recognition on the image. In addition, since the feature extraction methods in multiple feature extraction processes are independent of each other, the independence of various feature extraction methods is ensured, which is conducive to achieving more accurate and fine-grained pixel feature recognition results, and is conducive to improving the recognition accuracy of the image recognition process.

[0143] Figure 8A schematic diagram of the structure of another image recognition processing device provided in an embodiment of the present application, in Figure 7 Based on the embodiment shown, Figure 8 As shown,

[0144] The first processing unit 2 includes a first processing subunit 21, which is used to:

[0145] Repeat the following steps until the feature extraction method used by the data on each data node in the feature extraction process is determined for any subsequent data node in the feature extraction process:

[0146] Select any data node in the feature extraction and any data node after the data node to form a target data node pair, wherein the processing of the data of the previous data node in the target data node pair to the next data node in the target data node pair has multiple candidate feature extraction methods;

[0147] The first connection vector is formed by using the weight value of each candidate feature extraction method in the process from the previous data node to the next data node, wherein the weight value of each candidate feature extraction method is an arbitrary value;

[0148] The first connection vector is normalized to obtain a first eigenvector, in which only one element is 1 and the rest are 0;

[0149] It is determined that the candidate feature extraction method corresponding to element 1 is the feature extraction method used by the data of the previous data node to the next data node.

[0150] The first processing unit 2 further includes:

[0151] The second processing subunit 22 is used to modify each first feature vector in the next feature extraction process by using a graph convolution method according to the feature extraction method used in the previous feature extraction process of two adjacent feature extraction processes;

[0152] The third processing subunit 23 is used to determine the feature extraction method used in the next feature extraction process according to the corrected first feature vectors.

[0153] The second processing subunit 22 includes:

[0154] A first processing module 221 is used to determine a connection matrix between a first matrix and a second matrix, wherein the first matrix is ​​a matrix formed by a feature extraction method used in a previous feature extraction process, and the second matrix is ​​a matrix formed according to each first feature vector in a subsequent feature extraction process;

[0155] A second processing module 222 is used to determine the correlation value between the connection matrix and the first matrix, and add the correlation value to the second matrix to obtain a third matrix;

[0156] The third processing module 223 is used to determine each corrected first eigenvector according to the third matrix.

[0157] The second processing unit 3 comprises:

[0158] The fourth processing subunit 31 is used to perform feature extraction of at least one scale on the pixel feature recognition result according to the pixel feature recognition result of the image to be processed, so as to obtain a feature extraction result of at least one scale of the image to be processed;

[0159] The fifth processing subunit 32 is used to process the image to be processed according to the feature extraction result of at least one scale of the image to be processed.

[0160] The fourth processing subunit 31 includes:

[0161] The fourth processing module 311 is used to form a second connection vector by using weight values ​​of various candidate feature extraction methods corresponding to a feature extraction method of a scale, wherein the weight value of each candidate feature extraction method is an arbitrary value;

[0162] The fifth processing module 312 is used to perform normalization processing on the second connection vector to obtain a second eigenvector, in which only one element is 1 and the other elements are all 0;

[0163] The sixth processing module 313 is used to determine that the candidate feature extraction method corresponding to element 1 is the feature extraction method of the scale.

[0164] The first processing unit 2 further includes:

[0165] The sixth processing subunit 24 is used to process the output data of each feature extraction in k feature extractions before the first feature extraction by a set processing method to obtain k processing results;

[0166] The seventh processing subunit 25 is used to determine the input data for this feature extraction according to the k processing results.

[0167] This embodiment obtains an image to be processed; obtains the image to be processed; performs feature extraction n times on the image to be processed to obtain a pixel feature recognition result of the image to be processed, wherein the feature extraction methods used in each feature extraction process are not completely the same; based on the pixel feature recognition result of the image to be processed, performs feature extraction of at least one scale on the pixel feature recognition result to obtain a feature extraction result of at least one scale of the image to be processed; and performs content recognition processing on the image to be processed based on the feature extraction result of at least one scale of the image to be processed. In each feature extraction process, the feature extraction processing methods used by any two data nodes for the data of the previous data nodes are independent, which ensures the independence and diversity of the feature extraction methods for feature extraction using the basic unit, and the obtained pixel feature recognition results are highly accurate and have great reference value; using the graph convolutional network as a communication mechanism, in two adjacent feature extractions using the basic unit, the feature extraction method of the previous feature extraction is used to provide a decision for the feature extraction method of the next feature extraction, and by using the graph convolutional network to decide a more optimized feature extraction method, the recognition accuracy of the pixel feature recognition results can be effectively guaranteed; the feature extraction method used by each feature data node of the feature unit that constitutes the neural network is independent, which ensures the diversity of feature extraction methods when the feature unit is used to extract features of at least one scale of the pixel feature recognition results, which is conducive to improving the accuracy of the image recognition results; since the feature extraction methods used in each feature extraction process are not exactly the same, this can effectively avoid the exponential reduction of the feature extraction method, which is conducive to ensuring the diversity and extensiveness of the processing methods of the neural network, and can effectively ensure the image recognition performance of the neural network, and can effectively improve the image recognition accuracy.

[0168] Fig. 9 A schematic diagram of the structure of an image recognition processing device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the embodiment of the present application provides an image recognition processing device that can be used to perform Figure 1-Figure 2 The actions or steps of the image recognition processing device in the illustrated embodiment specifically include: a processor 901 , a memory 902 and a communication interface 903 .

[0169] The memory 902 is used to store computer programs.

[0170] Processor 901 is used to execute the computer program stored in memory 902 to implement Figure 1-Figure 4 The actions of the image recognition processing device in the illustrated embodiment will not be described in detail.

[0171] Alternatively, the image recognition processing device may further include a bus 904. The processor 901, the memory 902, and the communication interface 903 may be interconnected via the bus 904; the bus 904 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 904 may be divided into an address bus, a data bus, and a control bus. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0172] In the embodiments of the present application, the above embodiments can refer to and learn from each other, and the same or similar steps and nouns will not be repeated one by one.

[0173] Alternatively, part or all of the above modules may be implemented by being embedded in a chip of the image recognition processing device in the form of an integrated circuit. They may be implemented separately or integrated together. That is, the above modules may be configured as one or more integrated circuits for implementing the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0174] A computer-readable storage medium stores a computer program, which is executed by a processor to implement the above processing method.

[0175] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, an image recognition processing device or a data center by wired (for example, coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode to another website site, computer, image recognition processing device or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as an image recognition processing device, a data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0176] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0177] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0178] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Performing feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed; wherein the input data of each feature extraction is determined based on the output data of the previous k feature extractions; the input data of the mth feature extraction includes the image to be processed; wherein m is an integer greater than or equal to 1 and less than or equal to k, k is an integer greater than or equal to 1 and less than n, and n is an integer greater than 1; Each feature extraction process includes p data nodes, where p is an integer greater than 1; The data on a data node in each feature extraction process is determined based on the data of each data node before the data node in the current feature extraction process, which is processed by using the set feature extraction method and the input data of the current feature extraction; wherein the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in the feature extraction process, and the feature extraction methods used in each feature extraction process are not completely the same; the output data of each feature extraction process is the data on the last data node in the current feature extraction process; The image to be processed is processed according to the pixel feature recognition result of the image to be processed.

2. The method according to claim 1, characterized in that The image processing method is performed by a neural network, and each feature extraction process is performed by a layer of basic units in the neural network.

3. The method according to claim 1 or 2, characterized in that: The feature extraction method used in each feature extraction process is determined according to the following steps: Repeat the following steps until the feature extraction method used by the data on each data node in the feature extraction process is determined for any subsequent data node in the feature extraction process: Select any data node in the feature extraction and any data node after the data node to form a target data node pair, wherein there are multiple candidate feature extraction methods for processing the data of the previous data node in the target data node pair to the next data node in the target data node pair; Using the weight value of each candidate feature extraction method in the process from the previous data node to the next data node, a first connection vector is formed, wherein the weight value of each candidate feature extraction method is an arbitrary value; Normalizing the first connection vector to obtain a first eigenvector, wherein only one element of the first eigenvector is 1, and the other elements are all 0; The candidate feature extraction method corresponding to the determined element 1 is the feature extraction method adopted from the data of the previous data node to the next data node.

4. The method according to claim 3, characterized in that Before determining the feature extraction method used in the latter feature extraction process of two adjacent feature extraction processes, the method further includes: Using graph convolution to correct each first feature vector in the subsequent feature extraction process according to the feature extraction method used in the previous feature extraction process of the two adjacent feature extraction processes; The feature extraction method used in the subsequent feature extraction process is determined according to the corrected first feature vectors.

5. The method according to claim 4, characterized in that Using a graph convolution method to correct each first feature vector in the subsequent feature extraction process according to the feature extraction method used in the previous feature extraction process of the two adjacent feature extraction processes, including: Determine a connection matrix between a first matrix and a second matrix, wherein the first matrix is ​​a matrix formed by the feature extraction method used in the previous feature extraction process, and the second matrix is ​​a matrix formed according to each first feature vector in the next feature extraction process; Determine a correlation value between the connection matrix and the first matrix, and add the correlation value to the second matrix to obtain a third matrix; The corrected first eigenvectors are determined according to the third matrix.

6. The method according to any one of claims 1-2, characterized in that: Processing the image to be processed according to the pixel feature recognition result of the image to be processed includes: According to the pixel feature recognition result of the image to be processed, performing feature extraction of at least one scale on the pixel feature recognition result to obtain a feature extraction result of at least one scale of the image to be processed; The image to be processed is processed according to a feature extraction result of at least one scale of the image to be processed.

7. The method according to claim 6, characterized in that The following steps are used to determine the feature extraction method for each scale: Using weight values ​​of various candidate feature extraction methods corresponding to a feature extraction method of a scale to form a second connection vector, wherein the weight value of each candidate feature extraction method is an arbitrary value; Normalizing the second connection vector to obtain a second eigenvector, wherein only one element of the second eigenvector is 1, and the other elements are all 0; It is determined that the candidate feature extraction method corresponding to element 1 is the feature extraction method of the scale.

8. The method according to any one of claims 1-2, characterized in that: The input data for each feature extraction is determined based on the output data of the previous k feature extractions using the following steps: The output data of each feature extraction in k feature extractions before one feature extraction is processed by a set processing method to obtain k processing results; The input data for this feature extraction is determined according to the k processing results.

9. An image processing device, characterized in that: include: An acquisition unit, used for acquiring an image to be processed; A first processing unit is used to perform feature extraction on the image to be processed n times to obtain a pixel feature recognition result of the image to be processed; wherein the input data of each feature extraction is determined according to the output data of the previous k feature extractions; the input data of the mth feature extraction includes the image to be processed; wherein m is an integer greater than or equal to 1 and less than or equal to k, k is an integer greater than or equal to 1 and less than n, and n is an integer greater than 1; Each feature extraction process includes p data nodes, where p is an integer greater than 1; The data on a data node in each feature extraction process is determined based on the data of each data node before the data node in the current feature extraction process, which is processed by using the set feature extraction method and the input data of the current feature extraction; wherein the feature extraction method used in a feature extraction process includes the feature extraction method used for the data on each data node in the feature extraction process, and the feature extraction methods used in each feature extraction process are not completely the same; the output data of each feature extraction process is the data on the last data node in the current feature extraction process; The second processing unit is used to process the image to be processed according to the pixel feature recognition result of the image to be processed.

10. The device according to claim 9, characterized in that The first processing unit includes a first processing sub-unit, which is used to: Repeat the following steps until the feature extraction method used by the data on each data node in the feature extraction process is determined for any subsequent data node in the feature extraction process: Select any data node in the feature extraction and any data node after the data node to form a target data node pair, wherein there are multiple candidate feature extraction methods for processing the data of the previous data node in the target data node pair to the next data node in the target data node pair; Using the weight value of each candidate feature extraction method in the process from the previous data node to the next data node, a first connection vector is formed, wherein the weight value of each candidate feature extraction method is an arbitrary value; Normalizing the first connection vector to obtain a first eigenvector, wherein only one element of the first eigenvector is 1, and the other elements are all 0; The candidate feature extraction method corresponding to the determined element 1 is the feature extraction method adopted from the data of the previous data node to the next data node.

11. The device according to claim 10, characterized in that The first processing unit further includes: The second processing subunit is used to modify each first feature vector in the next feature extraction process by using a graph convolution method according to the feature extraction method used in the previous feature extraction process of two adjacent feature extraction processes; The third processing subunit is used to determine the feature extraction method used in the next feature extraction process according to the corrected first feature vectors.

12. The device according to claim 11, characterized in that The second processing subunit includes: A first processing module, used to determine a connection matrix between a first matrix and a second matrix, wherein the first matrix is ​​a matrix formed by the feature extraction method used in the previous feature extraction process, and the second matrix is ​​a matrix formed according to each first feature vector in the next feature extraction process; A second processing module, used for determining a correlation value between the connection matrix and the first matrix, and adding the correlation value to the second matrix to obtain a third matrix; The third processing module is used to determine the corrected first eigenvectors according to the third matrix.

13. The device according to any one of claims 9 to 12, characterized in that: The second processing unit comprises: a fourth processing subunit, configured to perform feature extraction of at least one scale on the pixel feature recognition result of the image to be processed according to the pixel feature recognition result of the image to be processed, so as to obtain a feature extraction result of at least one scale of the image to be processed; The fifth processing subunit is used to process the image to be processed according to the feature extraction result of at least one scale of the image to be processed.

14. The device according to claim 13, characterized in that The fourth processing subunit comprises: A fourth processing module, configured to form a second connection vector by using weight values ​​of various candidate feature extraction methods corresponding to a feature extraction method of a scale, wherein the weight value of each candidate feature extraction method is an arbitrary value; a fifth processing module, configured to perform normalization processing on the second connection vector to obtain a second eigenvector, wherein only one element of the second eigenvector is 1, and the other elements are all 0; The sixth processing module is used to determine that the candidate feature extraction method corresponding to element 1 is the feature extraction method of the scale.

15. The device according to any one of claims 9 to 12, characterized in that: The first processing unit further includes: The sixth processing subunit is used to process the output data of each feature extraction in k feature extractions before the first feature extraction by a set processing method to obtain k processing results; The seventh processing subunit is used to determine the input data for this feature extraction according to the k processing results.

16. An image processing device, characterized in that: include: processor, memory and computer program; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

17. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images