Feature extraction method and device

Abstract

The embodiment of the invention relates to the field of artificial intelligence, and discloses a feature extraction method and device. The method comprises the steps that a to-be-processed object is obtained, a segmented object is obtained according to the to-be-processed object, and the segmented object comprises part of elements in the to-be-processed object; and representing the segmented object through the first vector, and representing a part of elements in the segmented object through the second vector. And performing feature extraction on the first vector to obtain a first feature, and performing feature extraction on the second vector to obtain a second feature. And fusing the at least two second features according to the first target weight to obtain a first fused feature. And performing fusion processing on the first feature and the first fusion feature to obtain a second fusion feature, the second fusion feature being used for obtaining the feature of the to-be-processed object, so that the extracted feature of the to-be-processed object can better represent the to-be-processed object, and then the performance of a model applying the feature extraction method can be improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202110223032.8, and the original application date is February 26, 2021. The entire content of the original application is incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and specifically relates to a method and device for feature extraction. Background Art

[0003] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a discipline about how to use cameras / video cameras and computers to obtain the data and information of the objects to be photographed that we need. Figuratively speaking, it is to install eyes (cameras / video cameras) and a brain (algorithms) on the computer to replace the human eye to identify, track, and measure targets, etc., so that the computer can perceive the environment. Since perception can be regarded as extracting information from sensory signals, computer vision can also be regarded as the science of studying how to make artificial systems "perceive" from images or multi-dimensional data. Generally speaking, computer vision uses various imaging systems to replace the visual organs to obtain input information, and then uses the computer to replace the brain to process and interpret these input information. The ultimate research goal of computer vision is to enable the computer to observe and understand the world through vision like a human being and have the ability to adapt to the environment independently.

[0004] With the development of computer vision, more and more tasks can be performed using visual perception models, including image classification, 2D detection, semantic segmentation, key point detection, linear object detection (such as lane line or stop line detection in autonomous driving technology), drivable area detection, scene recognition, etc. How to make the visual perception model better complete the target task and make the performance and effect of the visual perception model better is a problem that has received much attention. Summary of the Invention

[0005] This application provides a method and device for feature extraction, which can make the features of the object to be processed extracted better represent the object to be processed, and further improve the performance of the model applying the feature extraction method.

[0006] To solve the above technical problems, the embodiments of this application provide the following technical solutions:

[0007] In a first aspect, the present application provides a method for feature extraction, which may include: extracting features from a first vector through a first feature extraction model to obtain first features. The first vector is used to represent the first segmented object. The first segmented object may include some elements in the object to be processed. Wherein, the data type of the object to be processed may be image data, text data, and voice data. It can be understood that the segmented object to be processed includes some elements in the object to be processed. When the object to be processed is an image, some elements in the object to be processed refer to some pixels in the image; when the object to be processed is text or voice, some elements in the object to be processed refer to words or phrases in the text or voice. Extracting features from a second vector through a second feature extraction model to obtain a plurality of second features. The second vector is used to represent some elements in the first segmented object.

[0008] Fuse at least two second features according to a first target weight to obtain a first fused feature. The first target weight is determined according to a first parameter value, and the first target weight is positively correlated with the first parameter value. The first parameter value is used to represent the similarity between each of the at least two second features and a target second feature, where the target second feature is any one of the at least two second features, or the first target weight is a second parameter value, and the second parameter value includes at least one preset constant. The similarity between one or more second features and the target second feature can be measured in different ways. For example, it can be measured by the magnitude of the inner product between two second features. The larger the inner product between two second features, the higher the similarity between the two second features, and the greater the weight, that is, the influence of the two features on each other is large. For example, assume that the second features include feature A, feature B, and feature C. When the target second feature is feature B, assume that the inner product between feature A and feature B is greater than the inner product between feature A and feature C, then it represents that the similarity between feature A and feature B is greater, and the influence of feature A on feature B is greater, while the influence of feature C on feature B is small. The weights can be set to 0.9 and 0.1 respectively. Then, fusing at least two second features according to the first target weight can be understood as 0.9*A + B + 0.1*C, and this result represents a first fused feature. It should be noted that using the inner product to represent the similarity between features is just one way to measure the similarity between two features, and there are other ways to measure the similarity between two features. For example, a neural network model can be trained to obtain the similarity between two features after training. The first target weight may also be preset. For example, it can be set that the influence of each of the at least two second features on the target second feature is the same. Then, the average value of the target second feature and one or more other second features can be calculated and added to the target second feature. It should be noted that the above does not exhaust the ways to measure the influence of one or more second features on the target second feature. In addition to the several measurement methods listed above, other methods can also be adopted for measurement.

[0009] Fuse the first feature and the first fusion feature to obtain a second fusion feature, which is used to obtain the feature of the object to be processed. The second fusion feature is used to determine the final feature of the object to be processed. In a possible implementation, the second fusion feature output by the last feature extraction module among the multiple feature extraction modules in the first feature extraction model is used to determine the finally extracted feature of the object to be processed. For each segmented object, the last feature extraction module will output the corresponding second fusion feature, and the set of the second fusion features is the final feature of the object to be processed. In a possible implementation, weight the second fusion feature corresponding to each segmented object output by the last feature extraction module, and use the weighted result as the final feature of the object to be processed.

[0010] As can be seen from the solution provided by the first aspect, the second feature extraction model is used to establish the association relationship between elements, that is, the first fusion feature. Fuse the first fusion feature with the first feature, so that the extracted feature includes the association relationship between elements, and thus can better represent the feature of the object to be processed. The more information about the object to be processed that the extracted feature can represent, the more conducive it is for the model to analyze the object to be processed.

[0011] In a possible implementation manner of the first aspect, the method may further include: obtaining a third feature, which is obtained by performing feature extraction on a third vector through the first feature extraction model. The third vector is used to represent the second segmented object, and the second segmented object may include some elements in the object to be processed. Fusing the first feature and the first fusion feature to obtain a second fusion feature may include: fusing the first feature and the third feature according to a second target weight to obtain a third fusion feature. The second target weight is determined according to a third parameter value, and the third parameter value is used to represent the similarity between the third feature and the first feature, or the second target weight is a fourth parameter value, and the fourth parameter value includes at least one preset constant. Fuse the third fusion feature and the first fusion feature to obtain a second fusion feature. In this implementation manner, the second feature extraction model is used to establish the association relationship between the segmented objects. During the process of extracting the feature of the object to be processed, the association relationship between the segmented objects is retained, and the association relationship between elements is also retained, so that the extracted feature can better represent the feature of the object to be processed. Furthermore, the performance of the model applying this feature extraction method can be improved.

[0012] In a possible implementation of the first aspect, the first vector is specifically used to represent the first segmented object carrying the first position information, where the first position information is the position information of the first segmented object in the object to be processed. Taking the object to be processed as an image as an example, the first position information can be represented by the coordinate information of a single pixel or can be represented by the coordinate information of multiple pixels. For example, when the image to be processed is evenly segmented to obtain multiple image blocks, the position information of each image block can be represented by the coordinates of the pixel at the upper left corner of each image block. For another example, when each image block is a regular rectangle or square, the position information of each image block can be represented by the coordinates of the pixel at the upper left corner and the coordinates of the pixel at the lower right corner of each image block. The first position information can also be represented by an encoded vector. In this implementation, more information, that is, the first position information, is included in the first vector, enabling the first feature extraction model to obtain more information. The more information the first feature extraction model can obtain, the more conducive it is for the first feature extraction model to learn to better extract image features.

[0013] In a possible implementation of the first aspect, each second vector is specifically used to represent a part of the elements in the first segmented object carrying the second position information, where the second position information is the position information of the part of the elements in the first segmented object in the first segmented object. In this implementation, more information, that is, the second position information, is included in the second vector. The more information the second feature extraction can obtain, the more conducive it is for the second feature extraction to learn to better extract image features.

[0014] In a possible implementation of the first aspect, fusing the first feature and the first fused feature to obtain a second fused feature may include: concatenating the first feature and the first fused feature head-to-tail to obtain the second fused feature. In this implementation, a specific method for fusing the first feature and the first fused feature is given, increasing the diversity of the solution.

[0015] In a possible implementation of the first aspect, fusing the first feature and the first fused feature to obtain a second fused feature may include: performing a target operation on the first feature and the first fused feature to obtain the second fused feature, where the target operation may include at least one of addition or multiplication. In this implementation, a specific method for fusing the first feature and the first fused feature is given, increasing the diversity of the solution.

[0016] In a possible implementation of the first aspect, performing a target operation on the first feature and the first fusion feature to obtain a second fusion feature may include: when the first fusion feature includes multiple ones, performing a head-to-tail splicing process on the multiple first fusion features to obtain a spliced feature. Mapping the spliced feature to a feature with a target length, where the target length is determined according to the length of the first feature. Performing an addition process on the first feature and the feature with the target length to obtain the second fusion feature. In this implementation, a specific method for fusing the first feature and the first fusion feature is given, increasing the diversity of the solution.

[0017] In a possible implementation of the first aspect, fusing at least two second features according to a first target weight to obtain a first fusion feature may include: using the at least two second features as the input of a target model, and the output of the target model is the first fusion feature. The target model may include one of a self-attention network Transformer, a convolutional neural network CNN, or a recurrent neural network RNN. When the target model is Transformer, the first target weight is determined according to the inner product between each second feature among the at least two second features and a target second feature. When the target model is one of CNN or RNN, the first target weight is preset. In this implementation, several methods for obtaining the first fusion feature are given, increasing the diversity of the solution.

[0018] In a possible implementation of the first aspect, the object to be processed is a to-be-processed image. The first vector is specifically used to represent the first segmented image, and the first segmented image may specifically include some pixels in the to-be-processed image. The second vector is specifically used to represent some pixels in the first segmented image, and the second fusion feature is specifically used to obtain the feature of the to-be-processed image. In this implementation, the object to be processed is a to-be-processed image. During the process of extracting image features, the association relationship between image blocks and the association relationship between pixels (or pixel blocks) are retained, enabling the extracted image features to well capture the color features, texture features, shape features, and spatial relationship features of the image, etc., thereby improving the performance of the visual perception model.

[0019] Second aspect, the present application provides a feature extraction model, which may include a first feature extraction model and a second feature extraction model. The first feature extraction model is used to obtain a first feature, which is obtained by performing feature extraction on a first vector by the first feature extraction model. The first vector is used to represent the first segmented object, and the first segmented object may include some elements in the object to be processed. The second feature extraction model is used to obtain a plurality of second features, which are obtained by performing feature extraction on a second vector by the second feature extraction model. The second vector is used to represent some elements in the first segmented object. The second feature extraction model is further used to fuse at least two second features according to a first target weight to obtain a first fused feature. The first target weight is determined according to a first parameter value, and the first target weight is positively correlated with the first parameter value. The first parameter value is used to represent the similarity between each second feature among at least two second features and a target second feature, where the target second feature is any one of the at least two second features, or the first target weight is a second parameter value, and the second parameter value includes at least one preset constant. The first feature extraction model is further used to perform a fusion process on the first feature and the first fused feature to obtain a second fused feature, and the second fused feature is used to obtain the feature of the object to be processed.

[0020] In a possible implementation manner of the second aspect, the first feature extraction model is further used to: obtain a third feature, which is obtained by performing feature extraction on a third vector by the first feature extraction model. The third vector is used to represent the second segmented object, and the second segmented object may include some elements in the object to be processed. Fuse the first feature and the third feature according to a second target weight to obtain a third fused feature. Specifically, the first feature extraction model is used when the second target weight is determined according to a third parameter value, and the third parameter value is used to represent the similarity between the third feature and the first feature, or the second target weight is a fourth parameter value, and the fourth parameter value includes at least one preset constant. Perform a fusion process on the third fused feature and the first fused feature to obtain the second fused feature.

[0021] In a possible implementation manner of the second aspect, the first vector is specifically used to represent the first segmented object carrying first position information, and the first position information is the position information of the first segmented object in the object to be processed.

[0022] In a possible implementation manner of the second aspect, each second vector is specifically used to represent some elements in the first segmented object carrying second position information, and the second position information is the position information of some elements in the first segmented object in the first segmented object.

[0023] In a possible implementation of the second aspect, the first feature extraction model is specifically configured to: perform head-to-tail splicing on the first feature and the first fused feature to obtain a second fused feature.

[0024] In a possible implementation of the second aspect, the first feature extraction model is specifically configured to: perform a target operation on the first feature and the first fused feature to obtain a second fused feature, and the target operation may include at least one of addition or multiplication.

[0025] In a possible implementation of the second aspect, when the first fused feature may include multiple ones, the first feature extraction model is specifically configured to: perform head-to-tail splicing on the multiple first fused features to obtain a spliced feature. Map the spliced feature to a feature with a target length, and the target length is determined according to the length of the first feature. Perform an addition operation on the first feature and the feature with the target length to obtain a second fused feature.

[0026] In a possible implementation of the second aspect, the second feature extraction model is specifically configured to: use at least two second features as the input of a target model, and the output of the target model is the first fused feature. The target model may include one of a self-attention network Transformer, a convolutional neural network CNN, or a recurrent neural network RNN. When the target model is a Transformer, the first target weight is determined according to the inner product between each of the at least two second features and a target second feature. When the target model is one of CNN or RNN, the first target weight is preset.

[0027] In a possible implementation of the second aspect, the object to be processed is an image to be processed. The first vector is specifically used to represent the first segmented image, and the first segmented image may specifically include some pixels in the image to be processed. The second vector is specifically used to represent some pixels in the first segmented image, and the second fused feature is specifically used to obtain the feature of the image to be processed.

[0028] For the specific implementation steps of the second aspect of this application and various possible implementation manners, as well as the beneficial effects brought by each possible implementation manner, reference may be made to the descriptions in various possible implementation manners in the first aspect, and details are not repeated here.

[0029] In a third aspect, this application provides an image processing method, which may include: obtaining an image to be processed. Inputting the image to be processed into a visual perception model to extract image features through a feature extraction model that may be included in the visual perception model, and the feature extraction model is the feature extraction model described in the second aspect or any possible implementation manner of the second aspect. Performing visual perception on the image to be processed according to the image features.

[0030] In a possible implementation of the third aspect, visual perception of the image to be processed based on image features may include: classifying the image to be processed according to the image features to obtain a classification result of the image to be processed.

[0031] In a possible implementation of the third aspect, obtaining the image to be processed may include: obtaining the image to be processed through a sensor of a vehicle. Visual perception of the image to be processed based on image features may include: performing semantic segmentation on the image to be processed according to the image features to obtain the region where the target object is located in the image to be processed, and the target object may include one or more of a person, a vehicle, and a road surface.

[0032] In a possible implementation of the third aspect, obtaining the image to be processed may include: obtaining the image to be processed through a monitoring device. Visual perception of the image to be processed based on image features may include: if it is recognized according to the image features that the image to be processed may include a person, then recognizing the attributes of the person according to the image features, and the attributes may include one or more of gender, skin color, age, and clothing.

[0033] Fourth aspect, the present application provides an electronic device, which may include a processor, the processor is coupled with a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any possible implementation manner of the first aspect is implemented.

[0034] Fifth aspect, the present application provides a computer-readable storage medium, which may include a program, and when it runs on a computer, it causes the computer to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0035] Sixth aspect, the present application provides a circuit system, the circuit system may include a processing circuit, and the processing circuit is configured to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0036] Seventh aspect, the present application provides a computer program product, the computer program product may include instructions, and when the instructions are loaded and executed by an electronic device, the electronic device is caused to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0037] Eighth aspect, the present application provides a chip, the chip is coupled with a memory, and is used to execute the program stored in the memory to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0038] For the specific implementation steps of the fourth to eighth aspects of this application and various possible implementation manners, as well as the beneficial effects brought by each possible implementation manner, reference may be made to the descriptions in various possible implementation manners in the first aspect, and details will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 FIG. is a schematic structural diagram of an artificial intelligence entity framework provided by an embodiment of this application;

[0040] Figure 2 FIG. is an architecture diagram of a system provided by an embodiment of this application;

[0041] Figure 3 FIG. is a schematic flowchart of feature extraction for an image;

[0042] Figure 4 FIG. is a schematic flowchart of a feature extraction method provided by an embodiment of this application;

[0043] Figure 5 FIG. is a schematic flowchart of obtaining an element set provided by an embodiment of this application;

[0044] Figure 6 FIG. is a schematic flowchart of converting an image block into a vector representation provided by an embodiment of this application;

[0045] Figure 7 FIG. is a schematic diagram of a feature extraction model provided by an embodiment of this application;

[0046] Figure 8 FIG. is a schematic diagram of a feature extraction model provided by an embodiment of this application;

[0047] Figure 9 FIG. is a schematic flowchart of a feature extraction method provided by an embodiment of this application;

[0048] Figure 10 FIG. is a schematic diagram of a feature extraction model provided by an embodiment of this application;

[0049] Figure 11 FIG. is a schematic diagram of an application scenario of a feature extraction method provided by an embodiment of this application;

[0050] Figure 12 FIG. is a schematic diagram of an application scenario of a feature extraction method provided by an embodiment of this application;

[0051] Figure 13 FIG. is a schematic architecture diagram of an image classification model provided by an embodiment of this application;

[0052] Figure 14Experimental result graph of an image classification task using the model of a feature extraction method provided by this application;

[0053] Figure 15 A schematic structural diagram of an electronic device provided by an embodiment of this application;

[0054] Figure 16 Another schematic structural diagram of an electronic device provided by an embodiment of this application;

[0055] Figure 17 Another schematic structural diagram of an electronic device provided by an embodiment of this application;

[0056] Figure 18 A schematic structural diagram of a chip provided by an embodiment of this application. Detailed implementation manners

[0057] An embodiment of this application provides a feature extraction method and apparatus, and the solution provided by this application can improve the performance and effect of a visual perception model.

[0058] The following describes the embodiments of this application with reference to the accompanying drawings. Those of ordinary skill in the art will know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are equally applicable to similar technical problems.

[0059] To better understand the solution provided by this application, the overall working process of an artificial intelligence system is first described. Please refer to Figure 1 , Figure 1 shown is a schematic structural diagram of an artificial intelligence main framework. The above artificial intelligence main framework is described from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general processes of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes the refinement process of "data - information - knowledge - wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (provision and processing technology implementation) to the industrial ecosystem of the system.

[0060] (1) Infrastructure

[0061] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the external world, and is supported through the basic platform. It communicates with the external world through sensors; the computing power is provided by intelligent chips. As an example, the intelligent chips include hardware acceleration chips such as central processing unit (CPU), neural-network processing unit (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), and field programmable gate array (FPGA). The basic platform includes relevant platform guarantees and supports such as distributed computing frameworks and networks, and may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for computing.

[0062] (2) Data

[0063] The data at the upper layer of the infrastructure indicates the data sources in the field of artificial intelligence. The data involves graphics, images, voices, texts, and also involves Internet of Things data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0064] (3) Data Processing

[0065] Data processing usually includes data training, machine learning, deep learning, search, inference, decision-making, etc.

[0066] Among them, machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.

[0067] Inference refers to the process of simulating the intelligent reasoning method of humans in a computer or intelligent system, and using formal information to perform machine thinking and solve problems according to the inference control strategy. The typical function is search and matching.

[0068] Decision-making refers to the process of making decisions after the intelligent information has been inferred, and usually provides functions such as classification, sorting, and prediction.

[0069] (4) General Capabilities

[0070] After the data is processed by the above-mentioned data processing, some general capabilities can be further formed based on the results of the data processing. For example, it can be an algorithm or a general system. For example, image classification, personalized management of images, personalized management of battery charging, text analysis, computer vision processing, speech recognition, and so on.

[0071] (5) Intelligent Products and Industrial Applications

[0072] Intelligent products and industrial applications refer to the products and applications of artificial intelligence systems in various fields. It is the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes the implementation of applications. Its application fields mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, autonomous driving, smart cities, etc.

[0073] Embodiments of the present application may be applied to multiple application scenarios in the above various fields. For example, it can be applied to the application scenario of natural language search to improve the accuracy of natural language search; it can also be applied to the application scenario of machine translation to make the translation results more accurate; it can also be applied to the application scenario of multi-turn dialogue to improve the efficiency of human-computer communication. Embodiments of the present application are mainly applied to application scenarios related to the field of computer vision in the above various fields. For example, embodiments of the present application can be applied to application scenarios such as face recognition, image classification, object detection, semantic segmentation, key point detection, linear object detection (such as lane line or stop line detection in autonomous driving technology), drivable area detection, scene recognition, etc. As an example, it can be specifically applied to the application scenario of autonomous driving. An autonomous vehicle obtains environmental images around the vehicle through a camera. The images obtained by the camera are segmented to segment the areas where different objects such as the road surface, roadbed, vehicle, and pedestrian are located in the image, so as to keep the vehicle driving in the correct area. In the field of autonomous driving, the accuracy of image segmentation is crucial for the safety of vehicle driving. Through the solution provided by the present application, the accuracy of image segmentation in the field of autonomous driving can be improved. As another example, embodiments of the present application can be applied to the field of intelligent monitoring. In the field of intelligent monitoring, pedestrian attribute recognition based on the images obtained by monitoring devices is a key task. The pedestrian attribute recognition task needs to identify common attributes of pedestrians, such as gender, age, hair, clothes, dressing, etc. This requires that the image features can represent more image information, such as carrying more detailed information of the image. The image features can be obtained by inputting the images obtained by the monitoring device into a feature extraction model, and the feature extraction model extracts the image features of the images. It should be noted that in this application, the feature extraction model is sometimes also referred to as the feature extraction module, and the two mean the same thing. For example, in the example of the field of intelligent monitoring, the images obtained by the monitoring device are input into a target model, which is used to perform the pedestrian attribute recognition task. The target model includes a feature extraction module, and the feature extraction module extracts the image features so that the target model can identify the pedestrian attributes according to the extracted image features. Through the solution provided by the embodiments of the present application, the performance of the feature extraction model can be improved, so that the extracted image features can better represent the image information. The more image information that the image features can represent, the more beneficial it is to improve the accuracy of visual analysis tasks. For the task of pedestrian attribute recognition, the more beneficial it is to improve the accuracy of pedestrian attribute recognition.

[0074] It should be understood that the application scenarios of the embodiments of the present application are not exhausted here. In the above-mentioned various scenarios, the feature extraction method provided by the embodiments of the present application can be adopted to improve the performance of the feature extraction model.

[0075] To facilitate the understanding of this solution, first in combination with Figure 2An introduction to a system provided by an embodiment of the present application is given. Please refer to Figure 2 , Figure 2 which is an architecture diagram of a system provided by an embodiment of the present application. In Figure 2 , the system 200 includes an execution device 210, a training device 220, a database 230, and a data storage system 240.

[0076] In the training phase, a training data set is stored in the database 230. The database 230 can be embodied in any form of storage medium, not limited to a traditional database. There can be multiple training samples in the training data set. The present application does not limit the data type of the training samples. For example, the training samples can be image data, or the training samples can be voice data, or the training samples can be text data. It should be noted that usually, the data types of the training samples included in the training data set are the same. The training device 220 generates a first machine learning model / rule 201 and iteratively trains the first machine learning model / rule 201 using the training data set in the database to obtain a mature first machine learning model / rule 201. When the training samples are image data, the present application also refers to the first machine learning model / rule 201 as a visual perception model. Taking the training samples as image data as an example, how to iteratively train the first machine learning model / rule 201 to obtain a mature first machine learning model / rule 201 is described. When the image data is used as the input of the first machine learning model / rule 201, the first machine learning model / rule 201 extracts the image features of the image data through a feature extraction model, and the first machine learning model / rule 201 iteratively trains the first machine learning model / rule 201 through the extracted image features. The training device can use the first machine learning model / rule 201 to train the data to obtain a mature first machine learning model / rule 201. The operation of each layer of the first machine learning model / rule 201 can be described by a mathematical expression as follows: The operation of each layer in a deep neural network from a physical level can be understood as completing the transformation from the input space (a set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimension increase / dimension decrease; 2. Magnification / minification; 3. Rotation; 4. Translation; 5. "Bending". Among them, the operations of 1, 2, and 3 are performed by Completed. The operation of 4 is completed by +b, and the operation of 5 is implemented by a(). The reason for using the word "space" here is that the objects to be classified are not individual things, but a class of things. Space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in this vector represents the weight value of a neuron in this layer of the neural network. This vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training the first machine learning model / rule 201 is to finally obtain the weight matrix of all layers of the trained first machine learning model / rule 201 (a weight matrix formed by vectors W of many layers). Therefore, the training process of the first machine learning model / rule 201 is essentially a process of learning the way to control space transformation, and more specifically, learning the weight matrix.

[0077] Because it is hoped that the output of the first machine learning model / rule 201 is as close as possible to the value that is really wanted to be predicted. Among them, the value that is really wanted to be predicted is related to the training objective of the first machine learning model / rule 201 or the task that the first machine learning model / rule 201 needs to complete. For example, if the first machine learning model / rule 201 is used for an image classification task, then the output of the first machine learning model / rule 201 is as close as possible to the true image classification result. It should be noted that this application focuses on researching how to make the features extracted by the first machine learning model / rule 201 better represent the information of the object to be processed. As for what specific task the first machine learning model / rule 201 performs based on the extracted features, this application does not make any limitations. In order to make the output of the first machine learning model / rule 201 as close as possible to the value that is really wanted to be predicted, the weight vector of each layer of the neural network can be updated by comparing the predicted value of the current network and the value that is really wanted, and then according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is too high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network can predict the value that is really wanted. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function. They are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the first machine learning model / rule 201 becomes a process of minimizing this loss as much as possible.

[0078] During the inference phase, the execution device 210 can call data, code, etc. in the data storage system 240, or store data, instructions, etc. in the data storage system 240. The data storage system 240 can be configured in the execution device 210 or be an external memory of the execution device 210. The execution device 210 can call a mature first machine learning model / rule 201 to extract features of the object to be processed and perform specific tasks based on the extracted features of the object to be processed. Among them, the data type of the object to be processed is generally the same as that of the training samples. The specific task is determined according to the training task in the training phase. For example, in the training phase, the first machine learning model / rule 201 is iteratively trained using the training data set in the database, so that the mature first machine learning model / rule 201 can extract features from an image and perform an image classification task based on the extracted features. Then in the inference phase, the execution device 210 can call the mature first machine learning model / rule 201 to extract features of the image and perform an image classification task based on the extracted image features.

[0079] In some embodiments of the present application, for example Figure 2 in, the "user" can directly interact with the execution device 210, that is, the execution device 210 and the client device are integrated into the same device. As an example, in some application scenarios, the execution device 210 can be embodied as a terminal device, such as a mobile phone, a camera, a smart home, etc. Then in the inference phase, the user can input the object to be processed through the execution device 210. For example, the user takes a photo through the camera, and the image obtained by the camera is used as the input of the mature first machine learning model / rule 201. In some other application scenarios, the execution device 210 can specifically be an execution device configured with a display screen. Then in the inference phase, after the execution device 210 completes one (or multiple) tasks, it can display the output result of the first machine learning model / rule 201 to the user. For example, after the execution device 210 performs an image classification task, it displays the result of the image classification to the user. The execution device 210 can also be embodied in other forms, which will not be listed one by one here, but Figure 2 is only a schematic diagram of the architecture provided by the embodiments of the present invention, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation.

[0080] In some other embodiments of the present application, the execution device 210 and the client device can be separate and independent devices. The execution device 210 is configured with an input / output interface to interact with the client device. The "user" can input at least one task to the execution device 210 through the input / output interface of the client device, and the execution device 210 returns the processing result to the client device through the input / output interface.

[0081] During the iterative training of the first machine learning model / rule 201 and when applying the mature first machine learning model / rule 201 to execute tasks, a process of feature extraction for the object to be processed is involved. Therefore, the solution provided in this application can be executed either by the training device 220 or by the execution device 210.

[0082] Currently, the input requirements of some first machine learning models / rules 201 are one-dimensional vectors. For example, self-attention network (Transformer), long short term memory (LSTM) neural network, and gated recurrent unit networks (GRU). The input requirements of these models are one-dimensional vectors. However, the object to be processed is often a multi-dimensional tensor. For example, an image is usually a three-dimensional tensor. Therefore, the object to be processed needs to be preprocessed to convert the tensor into a vector before it can be used as the input of these models. The applicant found that some preprocessing solutions for the object to be processed would damage the internal structure of the object to be processed, resulting in the loss of detailed information in the features of the object to be processed, which is not conducive to the correct prediction of these models. Here, taking the object to be processed as an image and the first machine learning model / rule 201 as Transformer as an example, the defects of some solutions are described.

[0083] Refer to Figure 3 , some practices are to divide the image into multiple image patches. As Figure 3 shown, the image is divided into nine image patches, and each image patch includes 1 / 9 of the image, that is, includes 1 / 9 of the pixels in the image. For each image patch, the image patch is converted into a vector representation, and the image patch represented by the vector is used as the input of Transformer. Transformer is a neural network based on the self-attention mechanism. For an image patch, when Transformer extracts image features, it can establish the association relationship between this image patch and all input image patches. However, the applicant found that this method destroys the internal structure of the image, specifically manifested in that this method only considers the association relationship between image patches and does not consider the association relationship between pixels and pixels. After converting the image patch into a vector representation, some association relationships between pixels and pixels are lost. For example, originally adjacent pixels are no longer adjacent due to the conversion of the image patch into a vector representation, so the adjacent relationship between pixels and pixels is lost. In addition, if an attempt is made to divide the image patch small enough to solve this problem, new problems will arise. That is, the increase in the number of image patches will lead to a significant increase in the amount of calculation. The increase in the amount of calculation results in a decrease in the model training efficiency and a decrease in the prediction efficiency of the trained model.

[0084] To solve the above problems, an embodiment of the present application provides a feature extraction method, so that the first machine learning model / rule 201 includes at least two self-attention modules, where one self-attention module is used to establish the association relationship between image patches, and the other self-attention module is used to establish the association relationship between pixels and pixels, thereby improving the performance of the model.

[0085] Refer to Figure 4 , which is a schematic flowchart of a feature extraction method provided by an embodiment of the present application.

[0086] As Figure 4 shown, a feature extraction method provided by an embodiment of the present application may include the following steps:

[0087] 401. Perform a segmentation process on the object to be processed to obtain the segmented object to be processed.

[0088] The data type of the object to be processed may be image data (hereinafter simply referred to as image), text data (hereinafter simply referred to as text), and voice data (hereinafter simply referred to as voice). It can be understood that the segmented object to be processed includes some elements in the object to be processed. When the object to be processed is an image, some elements in the object to be processed refer to some pixels in the image; when the object to be processed is text or voice, some elements in the object to be processed refer to words or phrases in the text or voice. In a preferred embodiment, the object to be processed in the present application is image data. In the following embodiments, a feature extraction method provided by the present application will be introduced by taking the object to be processed as image data as an example. For the sake of convenience of description, the segmented image is hereinafter referred to as an image patch, and each image patch includes some pixels in the image, and all image patches form a complete image.

[0089] In a possible implementation, the image may be evenly segmented so that the number of pixels included in each segmented image patch is the same. In a possible implementation, the image may also not be evenly segmented, so that the number of pixels included in each segmented image patch is not exactly the same. Specifically, the number of pixels included in some image patches is the same, the number of pixels included in some image patches is different, or the number of pixels included in all image patches is different. In addition, all the pixels included in each image patch may be adjacent pixels, or some pixels are adjacent pixels and some pixels are not adjacent pixels. The adjacent pixels refer to the spatial position relationship between pixels in the complete image being adjacent. In a preferred embodiment, the image may be evenly segmented, and all the pixels included in each image patch are adjacent pixels. For an image, when it is evenly segmented into n image patches, it can be understood with reference to Formula 1-1.

[0090]

[0091] Among them, X represents the image to be processed. X 1 to X n respectively represent the individual image blocks after segmentation. n is a positive integer greater than 1 and is used to represent the number of image blocks after segmentation. R represents a tensor, and the size of this tensor is n×p×p×3. Among them, the size of each image block is p×p×3. p×p can be used to represent two dimensions of the image block, and 3 represents another dimension, that is, the channel dimension. For example, if the pixel value of each image block includes an RGB (Red, Green, Blue) color value, then the channel dimension of the image block is 3, and the pixel value can be a long integer representing the color.

[0092] Segmenting the image to be processed into multiple image blocks helps to accelerate the progress of the model in extracting image features, that is, the model can process multiple image blocks in parallel and extract the image features of multiple image blocks simultaneously.

[0093] 402. For each processed object after segmentation, obtain multiple element sets.

[0094] Each element set includes some elements in the processed object after segmentation. For example, for each image block, obtain an element set, and each element set includes some pixels in each image block. For the sake of convenience in description, the following will refer to some pixels in the image block as pixel blocks.

[0095] For each image block, multiple pixel blocks can be obtained. The number of pixels included in any two of these multiple pixel blocks can be the same or different. In addition, the pixels included in each pixel block can be adjacent pixels or non - adjacent pixels. Exemplarily, it can be understood with reference to Formula 1 - 2.

[0096]

[0097] where i = 1, 2, …, n. n is a positive integer greater than 1 and is used to represent the number of image blocks after segmentation. m is a positive integer greater than 1 and is used to represent the number of pixel blocks included in an image block. c is used to represent the length of the vector corresponding to a pixel block.

[0098] Then for n image blocks, there are n groups of pixel blocks, which can be understood with reference to Formula 1 - 3.

[0099]

[0100] It should be noted that the pixels included in any two of the multiple pixel blocks may overlap.

[0101] Exemplarily, a method for obtaining a set of elements is given below. The set of elements can be obtained by means of image matrix column conversion (im2col). Im2col mainly converts the data contained in each window in the image data into column vectors, and finally arranges them into a new matrix by columns. The following combination will be used Figure 5 for illustration. As Figure 5 shown, each number represents a pixel, Figure 5 and the channel dimension of each pixel is not shown in []. By sliding a window through the image, where the size of the window can be customized. For example, Figure 5 in [] is a 3×3 window, and it can also be customized to other sizes, such as 2×2, 4×4, etc. The embodiments of the present application do not limit this. The step size of each window slide can also be customized. For example, the distance of each slide is 1 pixel, the distance of each slide is 2 pixels, etc. The embodiments of the present application do not limit this. Each time the window slides on the image block, all the pixels included in the window can be regarded as a pixel block. Since each pixel block has a channel dimension, after each pixel is expanded, each pixel corresponds to multiple positions in a column vector. For example, each pixel may include three channels of red, green, and blue. After each pixel block is expanded, it corresponds to 3 element positions in a column vector. Each pixel block can be converted into a row vector or a column vector. As Figure 5 shown in [], it shows the process of converting a pixel block into a column vector.

[0102] 403. Feature extraction is performed on the first vector by the first feature extraction model to obtain the first feature, and feature extraction is performed on the second vector by the second feature extraction model to obtain the second feature.

[0103] Among them, the first vector is used to represent the object to be processed after segmentation. For example, the first vector is used to represent the image block mentioned in steps 401 and 402. The second vector is used to represent some elements in the segmented object. For example, the second vector is used to represent the pixel block mentioned in steps 401 and 402.

[0104] The first feature extraction model and the second feature extraction model can be understood as multiple feature extraction modules in the first machine learning model / rule 201 mentioned above. For example, the first feature extraction model and the second feature extraction model can be CNN or RNN. The first feature extraction model includes multiple feature extraction modules, and the second feature extraction model includes multiple feature extraction modules. For one of the first feature extraction model or the second feature extraction model, the multiple feature extraction modules are connected end to end, and the output of the previous feature extraction module is used as the input of the next feature extraction module, so that the next feature extraction module can continue to perform feature extraction. Each feature extraction module has a specific weight matrix, which acts as a filter for extracting specific information from the input image matrix in image processing. By traversing the input with the weight matrix, the work of extracting specific features from the image is completed. For the feature extraction module of the current first feature extraction model, the output of the previous feature extraction module can be regarded as the first feature. Among them, the image features mainly include the color feature, texture feature, shape feature, and spatial relationship feature of the image, etc. The color feature is a global feature that describes the surface properties of the scene corresponding to the image or image region. Generally, the color feature is a feature based on pixel points, and at this time, all pixels belonging to the image or image region contribute individually. Since the color is insensitive to changes in the direction, size, etc. of the image or image region, the color feature cannot well capture the local features of the objects in the image. The texture feature is also a global feature, which also describes the surface properties of the scene corresponding to the image or image region. However, since the texture is only a property of the object surface and cannot fully reflect the essential attributes of the object, only using the texture feature cannot obtain high-level image content. Different from the color feature, the texture feature is not a feature based on pixel points, and it needs to perform statistical calculations in a region containing multiple pixel points. There are two types of representation methods for the shape feature. One is the contour feature, and the other is the region feature. The contour feature of the image mainly targets the outer boundary of the object, while the region feature of the image is related to the entire shape region. The spatial relationship feature refers to the mutual spatial position or relative direction relationship between multiple objects segmented from the image. These relationships can also be divided into connection / adjacency relationships, overlap / overlap relationships, and inclusion / containment relationships, etc. Usually, the spatial position information can be divided into two categories: relative spatial position information and absolute spatial position information. The former relationship emphasizes the relative situation between objects, such as the up-down, left-right relationships, etc., and the latter relationship emphasizes the distance and orientation between objects. It should be noted that the above-listed image features can be used as some examples of the specific features in the image. The image can also have other features, such as higher-level features: semantic features, which will not be elaborated here.

[0105] 404. Fuse at least two second features according to the first target weight to obtain a first fused feature.

[0106] For a feature extraction module in the second feature extraction model, the feature extraction module fuses at least two second features according to the first target weight to obtain a first fused feature. The purpose of obtaining the first fused feature is to establish the connection between pixel blocks. Among them, establishing the association relationship between pixel blocks can be understood as when extracting the image features of a pixel block, considering the influence of one or more other pixel blocks on this pixel block. The greater the influence of one or more other pixel blocks on this pixel block, the greater the weight; the smaller the influence of one or more other pixel blocks on this pixel block, the smaller the weight. The influence of one or more pixel blocks on this pixel block can be measured in different ways. For example, it can be measured by the similarity between the vectors corresponding to two pixel blocks. Specifically, it can be measured by the magnitude of the inner product between the vectors corresponding to two pixel blocks. The greater the inner product between the vectors corresponding to two pixel blocks, the higher the similarity between the two pixel blocks, and the greater the weight. For another example, a neural network model can also be trained, and the similarity between pixel blocks can be obtained through the neural network model. It is also possible to perform a preset operation on the vectors corresponding to two pixel blocks, and obtain the influence of other pixel blocks on this pixel block according to the result after the preset operation. For example, the average value of the vector corresponding to the pixel block to be processed and the vectors corresponding to the pixel blocks adjacent to the pixel block to be processed can be calculated, and the average value can be superimposed on the vector corresponding to the pixel block to be processed.

[0107] In a preferred embodiment, the second feature extraction model can be a neural network based on the self-attention mechanism. For example, the second feature extraction model can be a Transformer. When the second feature extraction model is a Transformer, after the second feature extraction model extracts features from the second vector, a first fused feature can be obtained. Taking the second feature extraction model as a Transformer and the object to be processed as image data as an example, the process of extracting features from the second vector through the second feature extraction model to obtain the second feature or the first fused feature will be described below. Refer to Figure 7 , when the second feature extraction model is a Transformer, the multiple feature extraction modules in the second feature extraction model can specifically be multiple feature extraction blocks (blocks) for feature processing. The multiple blocks are connected end to end, and the output of the previous block is used as the input of the next block, so that the next block can continue to perform feature extraction. Among them, each block has a specific weight matrix, and its role in image processing is equivalent to a filter for extracting specific information from the input image matrix. By traversing the input through the weight matrix, the work of extracting specific features from the image can be completed.

[0108] During the process of extracting image features for each block, the correlation between pixels (or pixel blocks) can be established in various ways. When each block of the second feature extraction model extracts features from pixel blocks, self-attention calculation is performed on multiple pixel blocks, taking into account the influence of each pixel block on the currently processed pixel block. Continue to refer to Figure 7 , which is a schematic diagram of the Transformer architecture. A block generally includes a normalization processing module, which is used to normalize the input. The normalization processing can be understood as making the mean of the input data 0 and the standard deviation 1, so that the loss value decreases smoothly during each training process. The output of the normalization processing module can be regarded as the second feature. A block can also include a self-attention module, and the output of the normalization processing module is used as the input of the self-attention module. When a block is a block of the second feature extraction model, self-attention calculation is performed on multiple pixel blocks through the self-attention module to establish the correlation between pixel blocks. A block can also include another normalization processing module to normalize the output of the self-attention module, so that the loss value can decrease more smoothly during each training process. For the current block, the output of the previous block of the current block can be regarded as the second feature, the input of the previous block can be regarded as the second vector, the output of the previous block is used as the input of the current block, and the output of the current block is the first fusion feature. For the next block of the current block, the output of the current block can be regarded as the second feature, and the output of the next block is the first fusion feature. It should be noted that when the current block is the first block, the input of the current block is not the second feature, but only the second vector.

[0109] In a preferred embodiment, the first feature extraction model and the second feature extraction model are the models mentioned above whose input requirements are one-dimensional vectors. For example, the first feature extraction model can be one of Transformer, GRU, and LSTM, and the second feature extraction model can be one of Transformer, GRU, and LSTM. As mentioned above, the input requirements of these feature extraction models are one-dimensional vectors. Therefore, for these models, after obtaining the image blocks and pixel blocks through steps 401 and 402, the image blocks need to be converted into vector representations, and the image blocks represented by vectors are used as the input of the first feature extraction model. The pixel blocks are converted into vector representations, and the pixel blocks represented by vectors are used as the input of the second feature extraction model. Among them, there are various representations for converting image blocks into vector representations. Exemplarily, refer to Figure 6For understanding, the pixels at the head and tail of each row of the image patch can be concatenated. Since each pixel patch has a channel dimension, after each pixel is unfolded, each pixel corresponds to multiple positions in a column vector. For example, each pixel can include three channels of red, green, and blue. After each pixel patch is unfolded, it corresponds to 3 element positions in a column vector. Each image patch can be transformed into a row vector or a column vector. After the vectors corresponding to all the image patches are sorted row by row or column by column, a vector matrix of the image patches can be obtained. The way of transforming the pixel patch into a vector representation can be understood by referring to the way of transforming the image patch into a vector representation. For example, in the above example of obtaining the pixel patch through im2col, all the pixels included in each window are unfolded into a column vector, and multiple column vectors can be obtained, and then sorted column by column to obtain the vector matrix of the pixel patch. In addition, the first feature extraction model and the second feature extraction model may also have requirements for the size of the input vector. For example, only vectors of a preset length can be used as the input of the first feature extraction model and the second feature extraction model. Therefore, in some possible implementation manners, it is also necessary to perform mapping processing on the vectors after each image patch is transformed to map the vectors after the image patch is transformed into vectors of a preset length to meet the requirements of the first feature extraction model for the input; it is also necessary to perform mapping processing on the vectors after each pixel patch is transformed to map the vectors after the pixel patch is transformed into vectors of a preset length to meet the requirements of the second feature extraction model for the input. The first vector in this application is used to represent the object to be processed after segmentation. For example, the first vector is used to represent the image patches mentioned in steps 401 and 402. The second vector is used to represent some elements in the object after segmentation. For example, the second vector is used to represent the pixel patches mentioned in steps 401 and 402. When the first vector is used as the input of the first feature extraction model and the second vector is used as the input of the second feature extraction model, the first vector also meets the input requirements of the first feature extraction model, and the second vector also meets the input requirements of the second feature extraction model.

[0110] 405. Perform fusion processing on the first feature and the first fusion feature to obtain a second fusion feature, where the second fusion feature is used to obtain the feature of the object to be processed.

[0111] For a feature extraction module in the second feature extraction model, the solution provided in this application can implement the fusion processing of the first feature and the first fusion feature in multiple ways. The following will be described from two aspects: the fusion timing and the fusion method.

[0112] First, the fusion timing will be described: Refer to Figure 8In sub - figure a, the solution provided by this application enables the first machine learning model / rule 201 to include two feature extraction models, namely the first feature extraction model and the second feature extraction model. Among them, the second feature extraction model establishes the association relationship between pixels (or pixel blocks). Specifically, it can be understood with reference to step 403. The first feature extraction model can establish the association relationship between image blocks. Regarding how to establish the association relationship between image blocks, it can be understood by referring to how to establish the association relationship between pixel blocks. Specifically, when extracting the image features of an image block, the influence of one or more other image blocks on this image block is taken into consideration, which will not be repeated here. As mentioned above, the first feature extraction model includes multiple feature extraction modules. Among them, for the current feature extraction module, the output of the previous feature extraction module is used to obtain the input of the next feature extraction module, and the input of the previous feature extraction module can be regarded as the first vector. Continue to refer to Figure 8 In sub - figure a, in a possible implementation manner, the first feature output by the previous feature extraction module in the first feature extraction model and the first fusion feature output by the current feature extraction module of the second feature extraction model can be fused to obtain the second fusion feature, and the second fusion feature is used as the input of the current feature extraction module in the first feature extraction model. Refer to Figure 8 In sub - figure b, in a possible implementation manner, the first feature output by the previous feature extraction module in the first feature extraction model can be used as the input of the current feature extraction module in the first feature extraction model. After fusing the output of the current feature extraction module in the first feature extraction model and the first fusion feature output by the current feature extraction module in the second feature extraction model, the second fusion feature is obtained and used as the input of the next feature extraction module in the first feature extraction model.

[0113] The fusion method is further described: In a possible implementation manner, when there are multiple first fusion features, the multiple first fusion features can be concatenated end - to - end to obtain the concatenated feature. The concatenated feature is mapped to a feature with a target length, and the target length is determined according to the length of the first feature. If the length of the concatenated feature is the same as that of the first feature, the two can be directly added, and the first feature and the feature with the target length are added to obtain the second fusion feature. In a possible implementation manner, the first feature and the first fusion feature are concatenated end - to - end to obtain the second fusion feature. For example, the first feature and the concatenated feature are concatenated to obtain the second fusion feature. In a possible implementation manner, a target operation is performed on the first feature and the first fusion feature to obtain the second fusion feature, and the target operation includes at least one of addition or multiplication.

[0114] The second fusion feature is used to determine the final feature of the object to be processed. In a possible implementation, the second fusion feature output by the last feature extraction module among the multiple feature extraction modules in the first feature extraction model is used to determine the finally extracted feature of the object to be processed. For each image patch, the last feature extraction module outputs the corresponding second fusion feature, and the set of second fusion features is the final feature of the object to be processed. In a possible implementation, the second fusion feature corresponding to each image patch output by the last feature extraction module is weighted, and the result after the weighting process is used as the final feature of the object to be processed.

[0115] From Figure 4 the corresponding embodiment, it can be seen that in the process of extracting image features by the first machine learning model / rule 201 provided by the present application, the association relationship between image patches and the association relationship between pixels (or pixel blocks) and pixels (or pixel blocks) are retained, so that the image features extracted by the first machine learning model / rule 201 can well capture the color features, texture features, shape features, spatial relationship features, etc. of the image, thereby improving the performance of the first machine learning model / rule 201. In addition, it should be noted that Figure 4 the corresponding embodiment mainly takes image data as an example for illustration, but it should be clear that the solution provided by the present application is equally applicable to other types of data. For example, for text data, in the process of extracting text features by the first machine learning model / rule 201, the association relationship between text blocks and the association relationship between word blocks and word blocks are retained, so that the text features extracted by the first machine learning model / rule 201 can well capture the semantic features of the text. Wherein, a text block can be understood as including some elements in the text data to be processed, such as including some adjacent words in the text data to be processed. A word block can be understood as including some elements in the text block, such as including some adjacent words in the text block. The following embodiments are all described by taking image data as an example, and other types of data can be understood with reference to the processing flow of image data, which will not be repeated hereinafter.

[0116] In order to enable the image features extracted by the first machine learning model / rule 201 to well capture the color features, texture features, shape features, spatial relationship features, etc. of the image, the position information of the image patches and pixel blocks can also be retained in the process of the model extracting image features. The following will be described in conjunction with an embodiment.

[0117] Refer to Figure 9 , which is a schematic flowchart of a method for feature extraction provided by an embodiment of the present application.

[0118] As Figure 9As shown in the figure, a method for feature extraction provided by an embodiment of the present application may include the following steps:

[0119] 901. Perform a segmentation process on the object to be processed to obtain the segmented object to be processed.

[0120] 902. For each segmented object to be processed, obtain a plurality of element sets.

[0121] Steps 901 and 902 can be understood with reference to Figure 4 401 and 402 in the corresponding embodiments, and will not be repeated here.

[0122] 903. Fuse the first position information on the first vector and fuse the second position information on the second vector.

[0123] The first position information is the position information of the segmented object in the object to be processed. For example, the first position information is the position information of an image block in an image block, and the second position information is the position information of some elements in the segmented object in the segmented object. For example, the second position information is the position information of a pixel block in an image block.

[0124] Among them, the first position information can be represented by the coordinate information of one pixel or can be represented by the coordinate information of multiple pixels. For example, when the object to be processed is evenly segmented to obtain a plurality of image blocks, the position information of each image block can be represented by the coordinates of the pixel at the upper left corner of each image block. For another example, when each image block is a regular rectangle or square, the position information of each image block can be represented by the coordinates of the pixel at the upper left corner and the coordinates of the pixel at the lower right corner of each image block. It should be noted that the coordinates of the pixel at the upper left corner and the coordinates of the pixel at the lower right corner here are only for illustrative purposes to illustrate that the first position information can be represented by the coordinate information of one pixel or the coordinate information of multiple pixels, and does not represent a limitation of the solution provided by the present application.

[0125] The second position information can be represented by the coordinate information of one pixel or can be represented by the coordinate information of multiple pixels. In addition, since the pixels included in a pixel block may all be non-adjacent pixels, the position information in a pixel block can be represented by the coordinates of all the pixels included in the pixel block.

[0126] In addition to representing the first position information and the second position information through the coordinate information of pixels, the first position information and the second position information can also be represented through encoding vectors. Taking the first position information as an example, the first machine learning model / rule 201 may include a position encoding module. In the initial state, the position encoding module may randomly set a vector to represent the position information of each image patch. During the iterative training of the first machine learning model / rule 201, the parameters of the position encoding module can be updated according to the loss value, so that the vector encoded by the position encoding module to represent the position information of the image patch can be closer to the true position information of the image patch.

[0127] Fusing the first position information on the first vector and the second position information on the second vector can be understood as updating X in Formula 1-1 n and Y0 in Formula 1-3 i which can be understood with reference to Formulas 1-4 and 1-5.

[0128] X n ←X n +E position-patch (1-4)

[0129] Y0 i ←Y0 i +E position-pixel (1-5)

[0130] where E position-patch is used to represent the first position information, and E position-pixel is used to represent the second position information.

[0131] Figure 4 The corresponding embodiment introduces that in a possible implementation manner, the weight matrices of the respective feature extraction modules in the first feature extraction model are all 0. Then the first vector only carries the first position information. For example: If the weight matrices of the respective extraction modules in the first feature extraction model are all 0, then for the first feature extraction module in the first feature model, the input of the first feature extraction module is n first vectors, and all elements in each vector of the n first vectors have a value of 0. The n first vectors are fused with the first position information and used as the input of the first feature extraction module. In a possible implementation manner, the input of the first feature extraction module can also be n + 1 first vectors, and all elements in each vector of the n + 1 first vectors have a value of 0. For the n + 1 first vectors, n of them are used to fuse the first position information, and the remaining one vector is used to represent the weighted average of the position information corresponding to each first image patch.

[0132] 904. Extract features from the first vector fused with the first location information through the first feature extraction model to obtain the first feature, and extract features from the second vector fused with the second location information through the second feature extraction model to obtain the second feature.

[0133] 905. Fuse at least two second features according to the first target weight to obtain the first fused feature.

[0134] 906. Perform a fusion process on the first feature and the first fused feature to obtain the second fused feature, and the second fused feature is used to obtain the features of the object to be processed.

[0135] Steps 904 to 906 can be understood with reference to Figure 4 Steps 403 to 405 in the corresponding embodiment. The difference is that Figure 9 The corresponding embodiment can provide more information to both the first feature extraction model and the second feature extraction model, specifically providing the first location information and the second location information. The more information the first feature extraction model and the second feature extraction model can obtain, the more helpful it is for the first feature extraction model and the second feature extraction model to learn, so as to better extract image features.

[0136] Figure 4 And Figure 9 In the embodiment of, it is introduced that the solution provided by this application enables the first machine learning model / rule 201 to include the first feature extraction model and the second feature extraction model. During the process of extracting image features through the first machine learning model / rule 201, the association relationship between image blocks and the association relationship between pixels (or pixel blocks) are retained, so that the image features extracted by the first machine learning model / rule 201 can well capture the color features, texture features, shape features, and spatial relationship features of the image, etc., thereby improving the performance of the first machine learning model / rule 201. In some possible implementation manners, the first machine learning model / rule 201 may further include a larger number of first feature extraction models and a larger number of second feature extraction models. A specific embodiment will be described below.

[0137] Refer to Figure 10 , which is a schematic diagram of the architecture of a model provided by an embodiment of this application. The model may include multiple feature extraction models, such as Figure 10The feature extraction models 1, 2, and 3 shown in Figure 4 and Figure 9 The corresponding embodiments have been introduced in detail and will not be repeated here.

[0138] In a possible implementation, the image to be processed can be segmented multiple times. For example, in one segmentation, the image to be processed is segmented into 4 image patches. After preprocessing the 4 image patches to meet the requirements of the feature extraction model 1 for input, it is used as the input of the feature extraction model 1. In another segmentation, the image to be processed is segmented into 16 image patches. After preprocessing the 16 image patches to meet the requirements of the feature extraction model 2 for input, it is used as the input of the feature extraction model 2. In another segmentation, the image to be processed is segmented into 64 image patches. After preprocessing the 64 image patches to meet the requirements of the feature extraction model 3 for input, it is used as the input of the feature extraction model 3. It should be noted that in a possible implementation, multiple feature extraction models, such as the feature extraction models 1, 2, and 3, can be executed in parallel simultaneously.

[0139] Through the solution provided by the embodiments of the present application, the performance of the feature extraction model can be improved, so that the extracted image features can better represent the image information. The more image information that the image features can represent, the more beneficial it is to improve the accuracy of the visual analysis task. Here, taking the solution provided by the present application applied to several typical visual analysis tasks as an example, the solution provided by the present application will be introduced.

[0140] Refer to Figure 11, when the solution provided by this application is applied to the application scenario of autonomous driving, the mature first machine learning model / rule 201 can be deployed on the autonomous vehicle or on the cloud device. After the autonomous vehicle obtains the environmental image around the vehicle through the camera, the obtained image is input into the preprocessing module, so that the preprocessing model performs segmentation processing on the image to obtain image blocks and pixel blocks, and converts the obtained image blocks and pixel blocks into vectors that meet the input requirements of the first feature extraction model and the second feature extraction model. Among them, the preprocessing module can be regarded as a part of the mature first machine learning model / rule 201 or as a separate part. When the preprocessing module is a separate part, the preprocessing module can be deployed on the autonomous vehicle, while the mature first machine learning model / rule 201 can be deployed on the cloud device. The first machine learning model / rule 201 extracts features from the environmental image around the vehicle obtained by the camera through the first feature extraction model and the second feature extraction model. Since during the feature extraction process, both the association relationship between image blocks and the association relationship between pixel blocks are retained, the extracted image features can better represent the area where each object is located in the environment around the vehicle, which is beneficial for the semantic segmentation model in the first machine learning model / rule 201 to segment the environmental image around the vehicle obtained by the camera according to the extracted features, so as to segment the areas where different objects such as the road surface, roadbed, vehicle, and pedestrian are located from the image, thereby keeping the vehicle driving in the correct area.

[0141] Refer to Figure 12 , when the solution provided by this application is applied to the field of intelligent monitoring, the mature first machine learning model / rule 201 can be deployed on the intelligent monitoring device or on the cloud device. The monitoring device (such as through Figure 12The images obtained by cameras A, B, and C shown in the figure are input into the preprocessing module, so that the preprocessing module performs segmentation processing on the images to obtain image blocks and pixel blocks, and converts the obtained image blocks and pixel blocks into vectors that meet the input requirements of the first feature extraction model and the second feature extraction model. Among them, the preprocessing module can be regarded as a part of the mature first machine learning model / rule 201, or it can be regarded as a separate part. When the preprocessing module is a separate part, the preprocessing module can be deployed on the monitoring device, while the mature first machine learning model / rule 201 can be deployed on the cloud device. The first machine learning model / rule 201 extracts features from the images obtained by the intelligent monitoring device through the first feature extraction model and the second feature extraction model. Since during the feature extraction process, both the association relationship between image blocks and the association relationship between pixel blocks are retained, the extracted image features can better represent the features of the objects that appear within the sensing range of the intelligent monitoring device. For example, if pedestrian attribute recognition is performed on a person, the image features extracted by the solution provided in this application can better represent the attributes and detailed features of the pedestrian, which is beneficial for the attribute recognition model in the first machine learning model / rule 201 to recognize the pedestrian attributes of the images obtained by the intelligent monitoring device according to the extracted features. For example, it can recognize the gender, age, hair color, clothes, dressing style, etc. of the pedestrian. Among them, the results of pedestrian attribute recognition can be displayed on the terminal device or stored in the server.

[0142] To more intuitively understand the beneficial effects brought by this solution, the beneficial effects brought by the embodiments of this application are described below in combination with data. During the test, the first machine learning model / rule 201 is used to perform an image classification task. As Figure 13 shown, it is a schematic flowchart of performing an image classification task by the first machine learning model / rule 201. The first machine learning model / rule 201 includes multiple target feature extraction models, and each target feature extraction model includes a first feature extraction model and a second feature extraction model. Figure 13 Among them, L represents a positive integer. The first machine learning model / rule 201 may further include an image preprocessing module, or the image preprocessing module may also be a module independent of the first machine learning model / rule 201. The image preprocessing module performs segmentation processing on the images to obtain image blocks and pixel blocks, and converts the obtained image blocks and pixel blocks into vectors that meet the input requirements of the first feature extraction model and the second feature extraction model. In addition, in a possible implementation manner, the vectors corresponding to multiple image blocks that meet the input requirements of the first feature extraction model may also be weighted, and the result is also used as the input of the first feature extraction model. The relevant processes executed by the image preprocessing module and the target feature extraction model can refer to Figure 4 andFigure 9 Understand the relevant descriptions in the corresponding embodiments, which will not be repeated here. The first machine learning model / rule 201 may also include a multi-layer perceptron head (MLP head), which is used to perform an image classification task based on the output of the last target feature extraction model to output a classification result. For example, in Figure 13 the corresponding scenario, the output classification result is "house". The configuration information of the first machine learning model / rule 201 used in the first test is shown in Table 1.

[0143] Table 1:

[0144]

[0145] Parameter 1 is used to represent the number of feature extraction modules included in the feature extraction model, that is, the first feature extraction model includes 12 feature extraction modules, and the second feature extraction model includes 12 feature extraction modules. Parameter 2 is used to represent the requirement of the second feature extraction model for the input vector length. Parameter 3 is used to represent the number of heads (multi-head self-attention) in the self-attention module of the second feature extraction model. Parameter 4 is used to represent the requirement of the first feature extraction model for the input vector length. Parameter 5 is used for the number of heads (multi-head self-attention) in the self-attention module of the first feature extraction model. Parameter 6 is used to represent the total number of parameters in the first machine learning model / rule 201 that applies a feature method provided in this application. The unit of the number of parameters is million. Parameter 7 is used to represent the number of floating point operations (FLOPs), and the unit is billion.

[0146] The test data set is the ImageNet data set, and a test experiment for image classification is carried out on the ImageNet data set. The test results are shown through Figure 14 displayed. As Figure 14As shown, compared with several existing image classification models, in the process of extracting image features by the first machine learning model / rule 201 provided in this application, the association relationship between image patches and the association relationship between pixels (or pixel blocks) and pixels (or pixel blocks) are retained, so that the image features extracted by the first machine learning model / rule 201 can well capture the color features, texture features, shape features, spatial relationship features, etc. of the image, thereby improving the classification accuracy of the first machine learning model / rule 201. Specifically, when the computational amounts of several existing models and the first machine learning model / rule 201 applying the feature extraction method provided in this application are the same, the first machine learning model / rule 201 applying the feature extraction method provided in this application has a higher classification accuracy for images. In addition, the test results also show that compared with the existing image classification models, the first machine learning model / rule 201 applying the feature extraction method provided in this application has a smaller computational amount. In other words, the first machine learning model / rule 201 applying the feature extraction method provided in this application has higher efficiency. Specifically, when the classification accuracies of several existing models and the first machine learning model / rule 201 applying the feature extraction method provided in this application for images are the same, the first machine learning model / rule 201 applying the feature extraction method provided in this application requires a smaller computational amount.

[0147] The above introduced a feature extraction method provided in the embodiments of this application. Through the feature extraction method provided in this application, the features of the object to be processed extracted can better represent the object to be processed, thereby improving the performance of the model applying this feature extraction method.

[0148] It can be understood that in order to implement the above functions, relevant devices for implementing the above solutions are also provided below. These relevant devices include the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, combining the modules and algorithm steps of each example described in the embodiments disclosed in this article, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0149] Specifically refer to Figure 15 , Figure 15A schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include a first acquisition module 1501, a second acquisition module 1502, a first fusion module 1503, a second fusion module 1504, and a third fusion module 1505.

[0150] The first acquisition module 1501 is configured to acquire a first feature. The second acquisition module 1502 is configured to acquire a plurality of second features. The first feature is obtained by performing feature extraction on a first vector through a first feature extraction model. The first vector is used to represent a first segmented object, and the first segmented object includes some elements in the object to be processed. The second feature is obtained by performing feature extraction on a second vector through a second feature extraction model. The second vector is used to represent some elements in the first segmented object. The first fusion module 1503 is configured to fuse at least two second features according to a first target weight to obtain a first fusion feature. The first target weight is determined according to the influence of each second feature among at least two second features on a target second feature, and the target second feature is any one of the at least two second features. The second fusion module 1504 is configured to perform a fusion process on the first feature and the first fusion feature to obtain a second fusion feature, and the second fusion feature is used to obtain the feature of the object to be processed.

[0151] In a possible implementation manner, the first acquisition module 1501 is further configured to acquire a third feature. The third feature is obtained by performing feature extraction on a third vector through the first feature extraction model. The third vector is used to represent a second segmented object, and the second segmented object includes some elements in the object to be processed. The third fusion module 1505 is configured to fuse the first feature and the third feature according to a second target weight to obtain a third fusion feature. The second target weight is determined according to the influence of the third feature on the first feature. The second fusion module 1504 is specifically configured to perform a fusion process on the third fusion feature and the first fusion feature to obtain a second fusion feature.

[0152] In a possible implementation manner, the first vector is specifically used to represent the first segmented object carrying first position information, and the first position information is the position information of the first segmented object in the object to be processed.

[0153] In a possible implementation manner, each second vector is specifically used to represent some elements in the first segmented object carrying second position information, and the second position information is the position information of some elements in the first segmented object in the first segmented object.

[0154] In a possible implementation manner, the second fusion module 1504 is specifically configured to perform a head-to-tail splicing process on the first feature and the first fusion feature to obtain a second fusion feature.

[0155] In a possible implementation manner, the second fusion module 1504 is specifically configured to perform a target operation on the first feature and the first fusion feature to obtain a second fusion feature, and the target operation includes at least one of addition or multiplication.

[0156] In a possible implementation manner, when there are multiple first fusion features, the second fusion module 1504 is specifically configured to perform head-to-tail splicing processing on the multiple first fusion features to obtain a spliced feature. Map the spliced feature to a feature with a target length, where the target length is determined according to the length of the first feature. Perform an addition process on the first feature and the feature with the target length to obtain a second fusion feature.

[0157] In a possible implementation manner, the first fusion module 1503 is specifically configured to use at least two second features as the input of a target model, and the output of the target model is a first fusion feature. The target model includes one of a self-attention network Transformer, a convolutional neural network CNN, or a recurrent neural network RNN. When the target model is a Transformer, the first target weight is determined according to the inner product between each second feature of the at least two second features and a target second feature. When the target model is one of CNN or RNN, the first target weight is preset.

[0158] In a possible implementation manner, the object to be processed is an image to be processed. The first vector is specifically used to represent the first segmented image, and the first segmented image specifically includes some pixels in the image to be processed. The second vector is specifically used to represent some pixels in the first segmented image, and the second fusion feature is specifically used to obtain the feature of the image to be processed.

[0159] In a possible implementation manner, the electronic device may be Figure 2 the training device 220 described in Figure 2 or the execution device 210 described in

[0160] It should be noted that Figure 15 for the information interaction, execution process, etc. among the modules in the electronic merger shown in Figures 4 to 10 is based on the same concept as the corresponding method embodiments in this application. For the specific content, reference can be made to the descriptions in the method embodiments shown in the foregoing of this application, and details are not described herein again.

[0161] The embodiments of this application further provide an electronic device. Please refer to Figure 16 Figure 16 which is a schematic structural diagram of the electronic device provided by the embodiments of this application. The electronic device 1400 may be deployed with Figures 4 to 10 ​The first machine learning model / rule 201 described in [the reference], which includes a first feature extraction model and a second feature extraction model and is used to execute Figures 4 to 10 the corresponding steps in [the reference]. Specifically, the electronic device 1400 may vary greatly due to different configurations or performances. It may include one or more central processing units (CPUs) 1422 (for example, one or more processors) and a memory 1432, and one or more storage media 1430 (for example, one or more mass storage devices) that store application programs 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 may be transient storage or persistent storage. In one embodiment, the memory 1432 is a random access memory (RAM), which can directly exchange data with the central processing unit 1422, and is used to load data 1444 and application programs 1442 and / or the operating system 1441 for the central processing unit 1422 to directly run and utilize, and is usually used as a temporary data storage medium for the operating system or other running programs. The programs stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the electronic device. Further, the central processing unit 1422 may be configured to communicate with the storage medium 1430 and execute a series of instruction operations in the storage medium 1430 on the electronic device 1400.

[0162] The electronic device 1400 may further include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, and / or one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0163] It should be noted that the central processing unit 1422 is also used to execute Figures 4 to 10 the other steps executed by the first machine learning model / rule 201 in [the reference]. For the specific implementation manners and the beneficial effects brought by the steps executed by the central processing unit 1422 for Figures 4 to 10 the first machine learning model / rule 201 in the corresponding embodiment, reference can be made to Figures 4 to 10 the descriptions in the corresponding method embodiments, which will not be elaborated here one by one.

[0164] The embodiment of the present application further provides an electronic device. Please refer to Figure 17 , Figure 17A schematic structural diagram of an electronic device provided by an embodiment of the present application. The first machine learning model / rule 201 described in Figures 4 to 10 can be deployed on the electronic device 1700. The first machine learning model / rule 201 includes a first feature extraction model and a second feature extraction model for performing the corresponding steps in Figures 4 to 10 . Specifically, the electronic device 1700 includes: a receiver 1701, a transmitter 1702, a processor 1703, and a memory 1704 (where the number of processors 1703 in the electronic device 1700 can be one or more, Figure 17 taking one processor as an example). Among them, the processor 1703 can include an application processor 17031 and a communication processor 17032. In some embodiments of the present application, the receiver 1701, the transmitter 1702, the processor 1703, and the memory 1704 can be connected through a bus or other means.

[0165] The memory 1704 can include a read-only memory and a random access memory, and provide instructions and data to the processor 1703. A part of the memory 1704 can also include a non-volatile random access memory (NVRAM). The memory 1704 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof. Among them, the operation instructions can include various operation instructions for implementing various operations.

[0166] The processor 1703 controls the operation of the electronic device. In a specific application, the various components of the electronic device are coupled together through a bus system. Among them, the bus system can include a power bus, a control bus, a status signal bus, etc. in addition to a data bus. However, for the sake of clear illustration, all kinds of buses are referred to as the bus system in the figure.

[0167] The method disclosed in the embodiment of the present application above can be applied to or implemented by the processor 1703. The processor 1703 can be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 1703 or the instructions in the form of software. The above-mentioned processor 1703 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1703 can implement or execute each method, step and logic block diagram disclosed in the embodiment of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 1704, and the processor 1703 reads the information in the memory 1704 and combines its hardware to complete the steps of the above method.

[0168] The receiver 1701 can be used to receive input digital or character information, and generate signal inputs related to the relevant settings and function controls of the execution device. The transmitter 1702 can be used to output digital or character information through the interface; the transmitter 1702 can also be used to send instructions to the disk group through the above interface to modify the data in the disk group; the transmitter 1702 can also include a display device such as a display screen.

[0169] In one case, in the embodiment of the present application, the application processor 17031 is used to execute Figures 4 to 10 the method executed by the first machine learning model / rule 201 described in the corresponding embodiment in

[0170] For the specific implementation manner and the beneficial effects brought by the application processor 17031 executing Figures 4 to 10 the function of the first machine learning model / rule 201 in the corresponding embodiment, reference can be made to Figures 4 to 10 the descriptions in the corresponding method embodiments, which will not be elaborated here one by one.

[0171] It should be understood that the above is only an example provided by the embodiments of the present application, and the vehicle may have more or fewer components than those shown, two or more components may be combined, or different configurations of components may be implemented.

[0172] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0173] The execution device and training device provided by the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit to enable the chip to execute the Figures 4 to 10 method for feature extraction of the model described in the embodiments shown above. Optionally, the storage unit is a storage unit inside the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit outside the chip located in the radio access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0174] Specifically, please refer to Figure 18 , Figure 18 , which is a schematic structural diagram of a chip provided by the embodiments of the present application. The chip may be embodied as a neural network processor NPU 160, and the NPU 160 is mounted on the main CPU (Host CPU) as a coprocessor, and tasks are assigned by the Host CPU. The core part of the NPU is the arithmetic circuit 1603, and the arithmetic circuit 1603 is controlled by the controller 1604 to extract matrix data from the memory and perform multiplication operations.

[0175] In some implementations, the arithmetic circuit 1603 includes multiple processing units (Process Engine, PE) inside. In some implementations, the arithmetic circuit 1603 is a two-dimensional systolic array. The arithmetic circuit 1603 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1603 is a general matrix processor.

[0176] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory 1602 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory 1601 and performs matrix operations with matrix B, and the partial results or final results of the obtained matrix are stored in the accumulator 1608.

[0177] The unified memory 1606 is used to store input data and output data. The weight data is directly transferred to the weight memory 1602 through the Direct Memory Access Controller (DMAC) 1605. The input data is also transferred to the unified memory 1606 through the DMAC.

[0178] The Bus Interface Unit 1610 (BIU) is used for the instruction fetch memory 1609 to obtain instructions from the external memory, and is also used for the storage unit access controller 1605 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0179] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1606, or transfer the weight data to the weight memory 1602, or transfer the input data to the input memory 1601.

[0180] The vector calculation unit 1607 includes multiple arithmetic processing units, and in case of need, further processes the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolution / full connection layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of the feature plane, etc.

[0181] In some implementations, the vector calculation unit 1607 can store the processed output vector in the unified memory 1606. For example, the vector calculation unit 1607 can apply a linear function and / or a non-linear function to the output of the arithmetic circuit 1603, such as performing linear interpolation on the feature plane extracted by the convolutional layer, or for example, the vector of the accumulated value, to generate activation values. In some implementations, the vector calculation unit 1607 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as the activation input to the arithmetic circuit 1603, such as for use in subsequent layers in the neural network.

[0182] The instruction fetch buffer 1609 connected to the controller 1604 is used to store the instructions used by the controller 1604; the unified memory 1606, the input memory 1601, the weight memory 1602, and the instruction fetch buffer 1609 are all On-Chip memories. The external memory is private to this NPU hardware architecture.

[0183] Among them, the operations of each layer in the recurrent neural network can be executed by the arithmetic circuit 1603 or the vector calculation unit 1607.

[0184] Among them, the processor mentioned anywhere above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the method in the first aspect above.

[0185] The embodiment of the present application also provides a chip, which includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin, or a circuit, etc. The processing unit can execute the computer-executable instructions stored in the storage unit so that the chip executes the above Figures 4 to 10 The method described therein. Optionally, the storage unit is a storage unit inside the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip located in the radio access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc. Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0186] In an embodiment of the present application, a computer-readable storage medium is further provided. A program for training a model is stored in the computer-readable storage medium. When it runs on a computer, the computer is caused to execute the above-mentioned Figures 4 to 10 method described.

[0187] In an embodiment of the present application, a computer program product is further provided. When it runs on a computer, the computer is caused to execute the steps in the method described in the foregoing Figures 4 to 10 embodiment shown. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state disk (SSD)).

[0188] In an embodiment of the present application, a circuit system is further provided. The circuit system includes a processing circuit configured to execute the steps in the method described in the foregoing Figures 4 to 10 embodiment shown.

[0189] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of pure software or software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including application specific integrated circuits, dedicated CLUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be diverse, such as analog circuits, digital circuits or dedicated circuits, etc. However, for the present application, software program implementation is a better embodiment in more cases. Based on such an understanding, the technical solution of the present application, in essence or the part that makes a contribution to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disc of a computer, etc., and includes several instructions to enable a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of the present application. In addition, this computer software product can also be embodied in the form of controls, drivers, independent or downloadable software objects, etc.

[0190] The terms "first", "second", etc. in the description, claims and the above-mentioned drawings of this application are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order other than those illustrated or described here. In this application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules does not necessarily have to be limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps that appear in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules that appears in this application is a logical division. In actual implementation, there can be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the shown or discussed coupling or direct coupling or communication connection between each other can be through some ports. The indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. And the modules or sub-modules described as separate components can be physically separated or not, can be physical modules or not, or can be distributed to multiple circuit modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application.

Claims

1. A method for feature extraction, characterized in that Including: Obtain a first feature and a plurality of second features. The first feature is obtained by extracting features from a first vector through a first feature extraction model. The first vector is used to represent a first segmented object, and the first segmented object includes some elements in the object to be processed. The second feature is obtained by extracting features from a second vector through a second feature extraction model, and the second vector is used to represent some elements in the first segmented object; Fuse at least two of the second features according to a first target weight to obtain a first fused feature. The first target weight is determined according to a first parameter value, and the first parameter value is used to represent the similarity between each of the at least two second features and a target second feature; Perform a fusion process on the first feature and the first fused feature to obtain a second fused feature, and the second fused feature is used to obtain the feature of the object to be processed.

2. The method according to claim 1, characterized in that The method further includes: Obtain a third feature, where the third feature is obtained by extracting features from a third vector through the first feature extraction model, and the third vector is used to represent a second segmented object, and the second segmented object includes some elements in the object to be processed; The performing a fusion process on the first feature and the first fused feature to obtain a second fused feature includes: Fuse the first feature and the third feature according to a second target weight to obtain a third fused feature. The second target weight is determined according to a third parameter value, and the third parameter value is used to represent the similarity between the third feature and the first feature, or the second target weight is a fourth parameter value, and the fourth parameter value includes at least one preset constant; Perform a fusion process on the third fused feature and the first fused feature to obtain the second fused feature.

3. The method according to claim 1 or 2, characterized in that, The first vector is specifically used to represent the first segmented object carrying first position information, and the first position information is the position information of the first segmented object in the object to be processed.

4. The method according to claim 1 or 2, characterized in that, Each of the second vectors is specifically used to represent some elements in the first segmented object carrying second position information, and the second position information is the position information of some elements in the first segmented object in the first segmented object.

5. The method according to claim 1 or 2, characterized in that, The performing a fusion process on the first feature and the first fused feature to obtain a second fused feature includes: Perform a head-to-tail splicing process on the first feature and the first fused feature to obtain the second fused feature.

6. The method according to claim 1 or 2, characterized in that, The performing a fusion process on the first feature and the first fused feature to obtain a second fused feature includes: Perform a target operation on the first feature and the first fused feature to obtain the second fused feature, and the target operation includes at least one of addition or multiplication.

7. The method according to claim 6, wherein The performing a target operation on the first feature and the first fused feature to obtain the second fused feature includes: When the first fusion feature includes a plurality of features, perform head-to-tail splicing processing on the plurality of first fusion features to obtain a spliced feature; Map the spliced feature to a feature with a target length, where the target length is determined according to the length of the first feature; Perform an addition process on the first feature and the feature with the target length to obtain the second fusion feature.

8. The method according to claim 1 or 2, characterized in that, The fusing at least two of the second features according to a first target weight to obtain a first fusion feature includes: Use the at least two second features as inputs to a target model, and the output of the target model is the first fusion feature. The target model includes a self-attention network Transformer, and the first target weight is determined according to the inner product between each of the at least two second features and the target second feature.

9. The method according to claim 1 or 2, characterized in that, The object to be processed is an image to be processed. The first vector is specifically used to represent the first segmented image, and the first segmented image specifically includes some pixels in the image to be processed. The second vector is specifically used to represent some pixels in the first segmented image, and the second fusion feature is specifically used to obtain the feature of the image to be processed.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Obtain an image to be processed; Input the image to be processed into a visual perception model to extract image features through the feature extraction model included in the visual perception model; Perform visual perception on the image to be processed according to the image features.

11. The method according to claim 10, characterized in that, The performing visual perception on the image to be processed according to the image features includes: Classify the image to be processed according to the image features to obtain a classification result of the image to be processed.

12. The method according to claim 10, characterized in that The obtaining an image to be processed includes: Obtain the image to be processed through a sensor of a vehicle; The performing visual perception on the image to be processed according to the image features includes: Perform semantic segmentation on the image to be processed according to the image features to obtain the region where the target object is located in the image to be processed, and the target object includes one or more of a person, a vehicle, and a road surface.

13. The method according to claim 10, characterized in that, The obtaining an image to be processed includes: Obtain the image to be processed through a monitoring device; The performing visual perception on the image to be processed according to the image features includes: If it is recognized according to the image features that the image to be processed includes a person, then recognize the attributes of the person according to the image features, and the attributes include one or more of gender, age, and clothing.

14. An electronic device, characterized in that, It includes a processor, the processor is coupled to a memory, and the memory stores program instructions. When the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 13 is implemented.

15. A computer-readable storage medium, characterized in that, It includes a program, and when it runs on a computer, it causes the computer to execute the method according to any one of claims 1 to 13.

16. A circuit system, characterized in that, The circuit system includes a processing circuit configured to execute the method according to any one of claims 1 to 13.

17. A computer program product, characterized in that, The computer program product includes instructions that, when loaded and executed by an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 13.

18. A chip, characterized in that, The chip is coupled to the memory and is configured to execute a program stored in the memory to perform the method according to any one of claims 1 to 13.

19. A feature extraction device, characterized in that, comprising an acquisition unit and a processing unit; The acquisition unit is configured to acquire a first feature and a plurality of second features. The first feature is obtained by performing feature extraction on a first vector through a first feature extraction model. The first vector is used to represent a first segmented object, and the first segmented object includes some elements in the object to be processed. The second feature is obtained by performing feature extraction on a second vector through a second feature extraction model. The second vector is used to represent some elements in the first segmented object; The processing unit is configured to fuse at least two of the second features according to a first target weight to obtain a first fused feature. The first target weight is determined according to a first parameter value, and the first parameter value is used to represent the similarity between each of the at least two second features and a target second feature; The processing unit is further configured to perform a fusion process on the first feature and the first fused feature to obtain a second fused feature, and the second fused feature is used to obtain the feature of the object to be processed.

20. The apparatus according to claim 19, wherein The acquisition unit is further configured to acquire a third feature, which is obtained by performing feature extraction on a third vector through the first feature extraction model. The third vector is used to represent a second segmented object, and the second segmented object includes some elements in the object to be processed; The processing unit is specifically configured to: fuse the first feature and the third feature according to a second target weight to obtain a third fused feature. The second target weight is determined according to a third parameter value, and the third parameter value is used to represent the similarity between the third feature and the first feature, or the second target weight is a fourth parameter value, and the fourth parameter value includes at least one preset constant; perform a fusion process on the third fused feature and the first fused feature to obtain the second fused feature.

21. The device according to claim 19 or 20, characterized in that, The first vector is specifically used to represent the first segmented object carrying first position information, and the first position information is the position information of the first segmented object in the object to be processed.

Images