A method and device for feature extraction

Abstract

The embodiment of the present application relates to the field of artificial intelligence, and discloses a method and device for feature extraction. The method includes: obtaining an object to be processed, obtaining a segmented object according to the object to be processed, and the segmented object includes some elements in the object to be processed. The segmented object is represented by a first vector, and some elements in the segmented object are represented by a second vector. The first feature is extracted from the first vector to obtain a first feature, and the second feature is obtained after the feature extraction is performed on the second vector. At least two second features are fused according to a first target weight to obtain a first fused feature. The first feature and the first fused feature are fused to obtain a second fused feature, and the second fused feature is used to obtain the feature of the object to be processed, so that the extracted feature of the object to be processed can better characterize the object to be processed, thereby improving the performance of the model to which the feature extraction method is applied.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for feature extraction. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a science about how to use cameras / camcorders and computers to obtain the data and information of the objects we need. Figuratively speaking, it is to install eyes (cameras / camcorders) and brains (algorithms) on computers to replace human eyes to identify, track and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] With the development of computer vision, more and more tasks can be performed using visual perception models, including image classification, 2D detection, semantic segmentation, key point detection, linear object detection (such as lane line or stop line detection in autonomous driving technology), drivable area detection, scene recognition, etc. How to enable visual perception models to better complete target tasks and improve the performance and effect of visual perception models is a problem that has attracted much attention. Summary of the invention

[0004] The present application provides a method and device for feature extraction, which can enable the extracted features of an object to be processed to better characterize the object to be processed, thereby improving the performance of a model to which the feature extraction method is applied.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for feature extraction, which may include: extracting features from a first vector through a first feature extraction model to obtain a first feature. The first vector is used to represent the object after the first segmentation. The object after the first segmentation may include some elements in the object to be processed. Among them, the data type of the object to be processed may be image data, text data, and voice data. It can be understood that the segmented object to be processed includes some elements in the object to be processed. When the object to be processed is an image, some elements in the object to be processed refer to some pixels in the image; when the object to be processed is text or voice, some elements in the object to be processed refer to words or phrases in the text or voice. Feature extraction is performed on a second vector through a second feature extraction model to obtain multiple second features. The second vector is used to represent some elements in the object after the first segmentation.

[0007] At least two second features are fused according to a first target weight to obtain a first fused feature, the first target weight is determined according to a first parameter value, and the first target weight is positively correlated with the first parameter value. The first parameter value is used to represent the similarity between each second feature of at least two second features and the target second feature, the target second feature is any second feature of at least two second features, or the first target weight is a second parameter value, and the second parameter value includes at least one preset constant. The similarity between one or more second features and the target second feature can be measured in different ways, such as by the size of the inner product between the two second features. The larger the inner product between the two second features, the higher the similarity of the two second features, and the larger the weight, that is, the two features have a great influence on each other; for example, assuming that the second feature includes feature A, feature B and feature C, when the target second feature is feature B, assuming that the inner product between feature A and feature B is greater than the inner product between feature A and feature C, it means that the greater the similarity between feature A and feature B, the greater the influence of feature A on feature B, and the influence of feature C on feature B is very small. The weights can be set to 0.9 and 0.1 respectively, then the fusion processing of at least two second features according to the first target weight can be understood as 0.9*A+B+0.1*C, and this result represents a first fusion feature. It should be noted that expressing the similarity between features through inner products is only a way to measure the similarity between two features. The similarity between two features can also be measured in other ways, such as training a neural network model to obtain the similarity between the two features after the training neural network. The first target weight may also be preset, for example, each of the at least two second features can be set to have the same impact on the target second feature, then the target second feature and one or more other second features can be averaged, and the average value can be superimposed on the target second feature. It should be noted that the above does not exhaustively list the ways to measure the impact of one or more second features on the target second feature. In addition to the several measurement methods listed above, other methods can also be used for measurement.

[0008] The first feature and the first fused feature are fused to obtain a second fused feature, and the second fused feature is used to obtain the feature of the object to be processed. The second fused feature is used to determine the final feature of the object to be processed. In one possible implementation, the second fused feature output by the last feature extraction module among the multiple feature extraction modules in the first feature extraction model is used to determine the final extracted feature of the object to be processed. For each segmented object, the last feature extraction module will output the corresponding second fused feature, and the set of second fused features is the final feature of the object to be processed. In one possible implementation, the second fused feature corresponding to each segmented object output by the last feature extraction module is weighted, and the result after weighted processing is used as the final feature of the object to be processed.

[0009] It can be seen from the solution provided in the first aspect that the association relationship between elements is established through the second feature extraction model, that is, the first fusion feature. The first fusion feature is fused with the first feature so that the extracted feature includes the association relationship between elements, thereby better representing the features of the object to be processed. The more information that the extracted features of the object to be processed can represent, the more conducive it is for the model to analyze the object to be processed.

[0010] In a possible implementation of the first aspect, the method may further include: obtaining a third feature, the third feature being obtained after feature extraction of the third vector by the first feature extraction model. The third vector is used to represent the second segmented object, and the second segmented object may include some elements in the object to be processed. The first feature and the first fused feature are fused to obtain the second fused feature, which may include: fusing the first feature and the third feature according to the second target weight to obtain the third fused feature. The second target weight is determined according to a third parameter value, and the third parameter value is used to represent the similarity between the third feature and the first feature, or the second target weight is a fourth parameter value, and the fourth parameter value includes at least one preset constant. The third fused feature and the first fused feature are fused to obtain the second fused feature. In this embodiment, the association relationship between the segmented objects and the segmented objects is established by the first feature extraction model. In the process of extracting the features of the object to be processed, the association relationship between the segmented objects and the segmented objects is retained, and the association relationship between the elements and the elements is retained, so that the extracted features can better characterize the features of the object to be processed. In turn, the performance of the model to which the feature extraction method is applied can be prompted.

[0011] In a possible implementation of the first aspect, the first vector is specifically used to represent the first segmented object carrying the first position information, and the first position information is the position information of the first segmented object in the object to be processed. Taking the object to be processed as an image as an example, the first position information can be represented by the coordinate information of a pixel or can be represented by the coordinate information of multiple pixels. For example, when the image to be processed is evenly segmented to obtain multiple image blocks, the position information of each image block can be represented by the coordinates of the pixel in the upper left corner of each image block. For another example, when each image block is a regular rectangle or square, the position information of each image block can be represented by the coordinates of the pixel in the upper left corner and the coordinates of the pixel in the lower right corner of each image block. The first position information can also be represented by a coding vector. In this embodiment, the first vector includes more information, namely the first position information, so that the first feature extraction model can obtain more information. The more information the first feature extraction model can obtain, the more helpful it is for the first feature extraction model to learn to better extract image features.

[0012] In a possible implementation of the first aspect, each second vector is specifically used to represent a partial element in the first segmented object that carries the second position information, and the second position information is the position information of the partial element in the first segmented object in the first segmented object. In this embodiment, the second vector includes more information, namely, the second position information. The more information the second feature extraction can obtain, the more helpful it is for the second feature extraction to learn, so as to better extract image features.

[0013] In a possible implementation of the first aspect, fusing the first feature and the first fused feature to obtain the second fused feature may include: performing end-to-end splicing on the first feature and the first fused feature to obtain the second fused feature. In this implementation, a specific method for fusing the first feature and the first fused feature is provided, which increases the diversity of solutions.

[0014] In a possible implementation of the first aspect, fusing the first feature and the first fused feature to obtain the second fused feature may include: performing a target operation on the first feature and the first fused feature to obtain the second fused feature, where the target operation may include at least one of addition or multiplication. In this implementation, a specific method for fusing the first feature and the first fused feature is provided, which increases the diversity of the solutions.

[0015] In a possible implementation of the first aspect, target operation is performed on the first feature and the first fused feature to obtain the second fused feature, which may include: when the first fused feature may include multiple features, multiple first fused features are spliced ​​end to end to obtain the spliced ​​feature. The spliced ​​feature is mapped to a feature of a target length, and the target length is determined according to the length of the first feature. The first feature and the feature of the target length are added to obtain the second fused feature. In this embodiment, a specific method for fusing the first feature and the first fused feature is provided, which increases the diversity of the scheme.

[0016] In a possible implementation of the first aspect, at least two second features are fused according to a first target weight to obtain a first fused feature, which may include: taking at least two second features as input to a target model, the output of the target model being the first fused feature, the target model may include one of a self-attention network Transformer, a convolutional neural network CNN, or a recurrent neural network RNN, when the target model is a Transformer, the first target weight is determined based on the inner product between each second feature of the at least two second features and the target second feature, when the target model is one of CNN or RNN, the first target weight is preset. In this embodiment, several ways to obtain the first fused feature are given, which increases the diversity of the scheme.

[0017] In a possible implementation of the first aspect, the object to be processed is an image to be processed, the first vector is specifically used to represent the image after the first segmentation, the image after the first segmentation may specifically include some pixels in the image to be processed, the second vector is specifically used to represent some pixels in the image after the first segmentation, and the second fusion feature is specifically used to obtain features of the image to be processed. In this embodiment, the object to be processed is an image to be processed, and in the process of extracting image features, the association relationship between image blocks and image blocks is retained, and the association relationship between pixels (or pixel blocks) and pixels (or pixel blocks) is retained, so that the extracted image features can well capture the color features, texture features, shape features, spatial relationship features, etc. of the image, thereby improving the performance of the visual perception model.

[0018] In a second aspect, the present application provides a feature extraction model, which may include a first feature extraction model and a second feature extraction model. The first feature extraction model is used to obtain a first feature. The first feature is obtained by extracting a first vector through the first feature extraction model. The first vector is used to represent the object after the first segmentation. The object after the first segmentation may include some elements in the object to be processed. The second feature extraction model is used to obtain multiple second features. The second feature is obtained by extracting a second vector through the second feature extraction model. The second vector is used to represent some elements in the object after the first segmentation. The second feature extraction model is also used to fuse at least two second features according to a first target weight to obtain a first fused feature. The first target weight is determined according to a first parameter value, and the first target weight is positively correlated with the first parameter value. The first parameter value is used to represent the similarity between each second feature of at least two second features and a target second feature. The target second feature is any second feature of at least two second features, or the first target weight is a second parameter value, and the second parameter value includes at least one preset constant. The first feature extraction model is also used to fuse the first feature and the first fusion feature to obtain a second fusion feature, and the second fusion feature is used to obtain the feature of the object to be processed.

[0019] In a possible implementation of the second aspect, the first feature extraction model is also used to: obtain a third feature, the third feature is obtained after the first feature extraction model performs feature extraction on the third vector, the third vector is used to represent the object after the second segmentation, and the object after the second segmentation may include some elements in the object to be processed. The first feature and the third feature are fused according to the second target weight to obtain a third fused feature, the first feature extraction model is specifically used for the second target weight is determined according to the third parameter value, the third parameter value is used to represent the similarity between the third feature and the first feature, or the second target weight is a fourth parameter value, and the fourth parameter value includes at least one preset constant. The third fused feature and the first fused feature are fused to obtain a second fused feature.

[0020] In a possible implementation manner of the second aspect, the first vector is specifically used to represent the first segmented object carrying the first position information, where the first position information is position information of the first segmented object in the object to be processed.

[0021] In a possible implementation of the second aspect, each second vector is specifically used to represent some elements in the first segmented object that carry second position information, and the second position information is position information of some elements in the first segmented object in the first segmented object.

[0022] In a possible implementation manner of the second aspect, the first feature extraction model is specifically used to: perform end-to-end concatenation processing on the first feature and the first fusion feature to obtain the second fusion feature.

[0023] In a possible implementation manner of the second aspect, the first feature extraction model is specifically used to: perform a target operation on the first feature and the first fused feature to obtain a second fused feature, and the target operation may include at least one of addition or multiplication.

[0024] In a possible implementation of the second aspect, the first feature extraction model is specifically used to: when the first fused feature may include multiple features, perform head-to-tail splicing processing on the multiple first fused features to obtain a spliced ​​feature. Map the spliced ​​feature to a feature of a target length, where the target length is determined according to the length of the first feature. Add the first feature and the feature of the target length to obtain a second fused feature.

[0025] In a possible implementation of the second aspect, the second feature extraction model is specifically used to: take at least two second features as input of a target model, the output of the target model is a first fusion feature, the target model may include one of a self-attention network Transformer, a convolutional neural network CNN, or a recurrent neural network RNN, when the target model is a Transformer, the first target weight is determined based on the inner product between each second feature of the at least two second features and the target second feature, when the target model is one of CNN or RNN, the first target weight is preset.

[0026] In a possible implementation of the second aspect, the object to be processed is an image to be processed, the first vector is specifically used to represent the image after the first segmentation, the image after the first segmentation may specifically include some pixels in the image to be processed, the second vector is specifically used to represent some pixels in the image after the first segmentation, and the second fusion feature is specifically used to obtain features of the image to be processed.

[0027] For the specific implementation steps of the second aspect of the present application and various possible implementation methods, as well as the beneficial effects brought about by each possible implementation method, please refer to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one.

[0028] In a third aspect, the present application provides an image processing method, which may include: obtaining an image to be processed. Inputting the image to be processed into a visual perception model to extract image features through a feature extraction model that may be included in the visual perception model, wherein the feature extraction model is the feature extraction model described in the second aspect or any possible implementation of the second aspect. Visually perceiving the image to be processed according to the image features.

[0029] In a possible implementation manner of the third aspect, performing visual perception on the image to be processed according to the image features may include: classifying the image to be processed according to the image features to obtain a classification result of the image to be processed.

[0030] In a possible implementation of the third aspect, obtaining the image to be processed may include: obtaining the image to be processed through a sensor of a vehicle. Performing visual perception on the image to be processed according to image features may include: performing semantic segmentation on the image to be processed according to the image features to obtain an area where a target object is located in the image to be processed, and the target object may include one or more of a person, a vehicle, and a road surface.

[0031] In a possible implementation of the third aspect, obtaining the image to be processed may include: obtaining the image to be processed through a monitoring device. Visually perceiving the image to be processed according to image features may include: if it is recognized according to the image features that the image to be processed may include a person, identifying attributes of the person according to the image features, and the attributes may include one or more of gender, skin color, age, and clothing.

[0032] In a fourth aspect, the present application provides an electronic device that may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any possible implementation manner of the first aspect is implemented.

[0033] In a fifth aspect, the present application provides a computer-readable storage medium, which may include a program, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0034] In a sixth aspect, the present application provides a circuit system, which may include a processing circuit, and the processing circuit is configured to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0035] In a seventh aspect, the present application provides a computer program product, which may include instructions. When the instructions are loaded and executed by an electronic device, the electronic device executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0036] In an eighth aspect, the present application provides a chip, which is coupled to a memory and is used to execute a program stored in the memory to perform the method described in the first aspect or any possible implementation manner of the first aspect.

[0037] For the specific implementation steps of the fourth to eighth aspects of the present application and various possible implementation methods, as well as the beneficial effects brought about by each possible implementation method, please refer to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the structure of the artificial intelligence main framework provided in the embodiment of the present application;

[0039] Figure 2 An architecture diagram of a system provided in an embodiment of the present application;

[0040] Figure 3 It is a schematic diagram of a process of extracting features from an image;

[0041] Figure 4 A flowchart of a feature extraction method provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a process for obtaining an element set is provided for an embodiment of the present application;

[0043] Figure 6 A schematic diagram of a process for converting an image block into a vector representation provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of a feature extraction model provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of a feature extraction model provided in an embodiment of the present application;

[0046] Fig. 9 A flowchart of a feature extraction method provided in an embodiment of the present application;

[0047] Fig.10 A schematic diagram of a feature extraction model provided in an embodiment of the present application;

[0048] Fig.11 A schematic diagram of an application scenario of a feature extraction method provided in an embodiment of the present application;

[0049] Fig.12 A schematic diagram of an application scenario of a feature extraction method provided in an embodiment of the present application;

[0050] Fig.13 A schematic diagram of the architecture of an image classification model provided in an embodiment of the present application;

[0051] Fig.14A graph showing the experimental results of an image classification task using a model of a feature extraction method provided in this application;

[0052] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0053] Fig.16 Another structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0054] Fig.17 Another structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0055] Fig.18 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The embodiments of the present application provide a method and device for feature extraction. The solution provided by the present application can improve the performance and effect of the visual perception model.

[0057] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0058] In order to better understand the solution provided by this application, the overall workflow of the artificial intelligence system is first described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the main framework of artificial intelligence from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of artificial intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0059] (1) Infrastructure

[0060] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, as an example, the smart chip includes hardware acceleration chips such as central processing unit (CPU), neural-network processing unit (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), field programmable gate array (FPGA); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0061] (2) Data

[0062] The data on the upper layer of the infrastructure indicates the data source in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0063] (3) Data processing

[0064] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0065] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0066] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0067] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0068] (4) General ability

[0069] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, image classification, personalized image management, personalized battery charging management, text analysis, computer vision processing, speech recognition, etc.

[0070] (5) Smart products and industry applications

[0071] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, smart cities, etc.

[0072] The embodiments of the present application may be applied to multiple application scenarios in the above-mentioned various fields, such as the application scenario of natural language search to improve the accuracy of natural language search; it can also be applied to the application scenario of machine translation to make the translation result more accurate; it can also be applied to the application scenario of multi-round dialogue to improve the efficiency of human-computer communication. The embodiments of the present application are mainly applied to the application scenarios related to the field of computer vision in the above-mentioned various fields. For example, the embodiments of the present application can be applied to face recognition, image classification, target detection, semantic segmentation, key point detection, linear object detection (such as lane line or stop line detection in autonomous driving technology), drivable area detection, scene recognition and other application scenarios. As an example, it can be specifically applied to the application scenario of autonomous driving. The autonomous driving vehicle obtains the environmental image around the vehicle through the camera. The image obtained by the camera is segmented to segment the areas where different objects such as the road surface, roadbed, vehicle, pedestrian, etc. are located from the image, so as to keep the vehicle driving in the correct area. In the field of autonomous driving, the accuracy of image segmentation is crucial to the safety of vehicle driving. Through the solution provided by this application, the accuracy of image segmentation in the field of autonomous driving can be improved. As another example, the embodiments of the present application can be applied to the field of intelligent monitoring. In the field of intelligent monitoring, pedestrian attribute recognition based on images acquired by monitoring equipment is a key task. The pedestrian attribute recognition task needs to identify common attributes of pedestrians, such as gender, age, hair, clothes, and outfits. This requires that the image features can represent more image information, such as carrying more detailed information of the image, wherein the image features can be obtained by inputting the image acquired by the monitoring device into the feature extraction model, and extracting the image features of the image through the feature extraction model. It should be noted that the present application sometimes also refers to the feature extraction model as a feature extraction module, and the two have the same meaning. For example, in this example of the intelligent monitoring field, the image acquired by the monitoring device is input into the target model, and the target model is used to perform the pedestrian attribute recognition task. The target model includes a feature extraction module, and the image features are extracted by the feature extraction module, so that the target model recognizes the pedestrian attributes according to the extracted image features. Through the scheme provided in the embodiment of the present application, the performance of the feature extraction model can be improved, so that the extracted image features can better represent the image information. The more image information that the image features can represent, the more conducive to improving the accuracy of the visual analysis task. For the task of pedestrian attribute recognition, it is more conducive to improving the accuracy of pedestrian attribute recognition.

[0073] It should be understood that the application scenarios of the embodiments of the present application are not exhaustively listed here. In the aforementioned various scenarios, the feature extraction method provided by the embodiments of the present application can be adopted to improve the performance of the feature extraction model.

[0074] In order to facilitate the understanding of this solution, first combine Figure 2For an introduction to a system provided in an embodiment of the present application, please refer to Figure 2 , Figure 2 A system architecture diagram is provided for an embodiment of the present application. Figure 2 In the embodiment, the system 200 includes an execution device 210, a training device 220, a database 230 and a data storage system 240.

[0075] During the training phase, a training data set is stored in the database 230. The database 230 can be specifically expressed as a storage medium in any form, and is not limited to a database in the traditional sense. There can be multiple training samples in the training data set. The present application does not limit the data type of the training sample. For example, the training sample can be image data, or the training sample can be voice data, or the training sample can be text data. It should be noted that the data types of the training samples included in the training data set are usually the same. The training device 220 generates a first machine learning model / rule 201, and iteratively trains the first machine learning model / rule 201 using the training data set in the database to obtain a mature first machine learning model / rule 201. When the training sample is image data, the present application also refers to the first machine learning model / rule 201 as a visual perception model. Taking the training sample as image data as an example, how to iteratively train the first machine learning model / rule 201 to obtain a mature first machine learning model / rule 201 is explained. When the image data is used as the input of the first machine learning model / rule 201, the first machine learning model / rule 201 extracts the image features of the image data through the feature extraction model, and the first machine learning model / rule 201 iteratively trains the first machine learning model / rule 201 through the extracted image features. The training device can use the first machine learning model / rule 201 to train the data to obtain a mature first machine learning model / rule 201. The work of each layer of the first machine learning model / rule 201 can be expressed by mathematical expressions To describe: From a physical level, the work of each layer in a deep neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Zoom in / out; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are completed by W·, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. The vector W determines the spatial transformation from input space to output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training the first machine learning model / rule 201 is to finally obtain the weight matrix of all layers of the trained first machine learning model / rule 201 (the weight matrix formed by many layers of vectors W). Therefore, the training process of the first machine learning model / rule 201 is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0076] Because it is hoped that the output of the first machine learning model / rule 201 is as close as possible to the value that is really wanted to be predicted. The value that is really wanted to be predicted is related to the training target of the first machine learning model / rule 201 or the task that the first machine learning model / rule 201 needs to complete. For example, if the first machine learning model / rule 201 is used to perform an image classification task, the output of the first machine learning model / rule 201 is as close as possible to the actual image classification result. It should be noted that this application focuses on how to make the features extracted by the first machine learning model / rule 201 better represent the information of the object to be processed. As for what specific tasks the first machine learning model / rule 201 performs based on the extracted features, this application does not limit it. In order to make the output of the first machine learning model / rule 201 as close as possible to the value that is really wanted to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the two by comparing the predicted value of the current network and the really wanted target value (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, the weight vector is adjusted to make it predict lower, and it is adjusted continuously until the neural network can predict the really wanted target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the first machine learning model / rule 201 becomes a process of minimizing this loss as much as possible.

[0077] In the inference stage, the execution device 210 can call the data, code, etc. in the data storage system 240, or store the data, instructions, etc. in the data storage system 240. The data storage system 240 can be configured in the execution device 210, or it can be a memory outside the execution device 210. The execution device 210 can call the mature first machine learning model / rule 201 to extract the features of the object to be processed, and perform specific tasks according to the extracted features of the object to be processed. Among them, the data type of the object to be processed and the data type of the training sample are generally the same. The specific task is determined according to the training task in the training stage. For example, in the training stage, the first machine learning model / rule 201 is iteratively trained using the training data set in the database, so that the mature first machine learning model / rule 201 can extract features of the image and perform image classification tasks according to the extracted features. Then, in the inference stage, the execution device 210 can call the mature first machine learning model / rule 201 to extract features of the image, and perform image classification tasks according to the extracted image features.

[0078] In some embodiments of the present application, for example Figure 2 In the example, the "user" can interact directly with the execution device 210, that is, the execution device 210 and the client device are integrated into the same device. As an example, in some application scenarios, the execution device 210 can be a terminal device, such as a mobile phone, a camera, a smart home, etc. In the reasoning stage, the user can input the object to be processed through the execution device 210. For example, the user takes a photo through a camera, and the image obtained by the camera is used as the input of the mature first machine learning model / rule 201. In other application scenarios, the execution device 210 can be specifically expressed as an execution device equipped with a display screen. In the reasoning stage, after completing a task (or multiple tasks), the execution device 210 can display the output result of the first machine learning model / rule 201 to the user. For example, after the execution device 210 performs an image classification task, it displays the result of image classification to the user. The execution device 210 can also be expressed in other forms, which are not listed here one by one, but Figure 2 It is only a schematic diagram of the architecture provided by an embodiment of the present invention, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation.

[0079] In other embodiments of the present application, the execution device 210 and the client device may be separate devices. The execution device 210 is configured with an input / output interface to interact with the client device for data. The "user" may input at least one task to the execution device 210 through the input / output interface of the client device, and the execution device 210 may return the processing result to the client device through the input / output interface.

[0080] In the process of iteratively training the first machine learning model / rule 201, and in the process of applying the mature first machine learning model / rule 201 to perform tasks, the process of feature extraction of the object to be processed is involved. Therefore, the solution provided in this application can be executed by both the training device 220 and the execution device 210.

[0081] At present, the input requirement of some first machine learning models / rules 201 is a one-dimensional vector. For example, the input requirement of self-attention network (Transformer), long short term memory (LSTM) neural network and gated recurrent unit networks (GRU) is a one-dimensional vector. However, the object to be processed is often a multi-dimensional tensor, such as an image is usually a three-dimensional tensor. Therefore, the object to be processed must be preprocessed and the tensor must be converted into a vector before it can be used as the input of these models. The applicant has found that some schemes for preprocessing the object to be processed will destroy the internal structure of the object to be processed, resulting in the loss of detailed information of the extracted features of the object to be processed, which is not conducive to the correct prediction of these models. The following takes the case where the object to be processed is an image and the first machine learning model / rule 201 is a Transformer as an example to illustrate the defects of some schemes.

[0082] See also Figure 3 , some approaches are to divide the image into multiple image blocks. Figure 3 As shown, the image is divided into nine image blocks, each of which includes 1 / 9 of the image, that is, 1 / 9 of the pixels in the image. For each image block, the image block is converted into a vector representation, and the image block represented by the vector is used as the input of the Transformer. Transformer is a neural network based on the self-attention mechanism. For an image block, when extracting image features, the Transformer can establish an association relationship between this image block and all input image blocks. However, the applicant found that this method causes the internal structure of the image to be destroyed, which is specifically manifested in that this method only considers the association relationship between image blocks and image blocks, but does not consider the association relationship between pixels and pixels. After the image block is converted into a vector representation, the association relationship between some pixels and pixels is lost. For example, adjacent pixels are no longer adjacent to adjacent pixels due to the conversion of image blocks into vector representations, and the adjacent relationship between pixels and pixels is lost. In addition, if you try to divide the image block into a small enough size to solve this problem, it will cause new problems. That is, the increase in the number of image blocks will greatly increase the amount of calculation, and the increase in the amount of calculation will lead to a decrease in the efficiency of model training and the prediction efficiency of the trained model.

[0083] In order to solve the above problems, an embodiment of the present application provides a feature extraction method, so that the first machine learning model / rule 201 includes at least two self-attention modules, one of which is used to establish an association relationship between image blocks and image blocks, and the other self-attention module is used to establish an association relationship between pixels and pixels, thereby improving the performance of the model.

[0084] See also Figure 4 , which is a flow chart of a feature extraction method provided in an embodiment of the present application.

[0085] like Figure 4 As shown, a feature extraction method provided in an embodiment of the present application may include the following steps:

[0086] 401. Split the object to be processed to obtain the split object to be processed.

[0087] The data type of the object to be processed can be image data (hereinafter referred to as image), text data (hereinafter referred to as text) and voice data (hereinafter referred to as voice). It can be understood that the object to be processed after segmentation includes some elements in the object to be processed. When the object to be processed is an image, some elements in the object to be processed refer to some pixels in the image; when the object to be processed is text or voice, some elements in the object to be processed refer to words or phrases in the text or voice. In a preferred embodiment, the object to be processed in the present application is image data. In the following embodiments, a method for feature extraction provided in the present application will be introduced by taking the object to be processed being image data as an example. For the sake of ease of description, the segmented image will be referred to as an image block below, each image block includes some pixels in the image, and all image blocks constitute a completed image.

[0088] In one possible implementation, the image may be evenly segmented so that the number of pixels included in each image block after segmentation is the same. In one possible implementation, the image may not be evenly segmented so that the number of pixels included in each image block after segmentation is not exactly the same. Specifically, the number of pixels included in some image blocks is the same, the number of pixels included in some image blocks is different, or the number of pixels included in all image blocks is different. In addition, the pixels included in each image block may all be adjacent pixels, or some pixels may be adjacent pixels, and some pixels may not be adjacent pixels. Adjacent pixels refer to the spatial position relationship between pixels in the complete image being adjacent. In a preferred implementation, the image may be evenly segmented, and all pixels included in each image block are adjacent pixels. For an image, it is evenly segmented into n image blocks, which can be understood with reference to formula 1-1.

[0089]

[0090] Where X represents the image to be processed. 1 To X n Respectively represent each image block after segmentation, n is a positive integer greater than 1, which is used to represent the number of image blocks after segmentation. R represents a tensor, the size of which is n×p×p×3, where the size of each image block is p×p×3, p×p can be used to represent two dimensions of the image block, and 3 represents another dimension, namely the channel dimension. For example, each image block includes an image pixel value which can be a red, green, and blue (RGB) color value, then the channel dimension of the image block is 3, and the pixel value can be a long integer representing the color.

[0091] Dividing the image to be processed into multiple image blocks helps speed up the model's progress in extracting image features, that is, the model can process multiple image blocks in parallel and extract image features of multiple image blocks at the same time.

[0092] 402. For each segmented object to be processed, obtain multiple element sets.

[0093] Each element set includes some elements in the segmented object to be processed. For example, for each image block, an element set is obtained, and each element set includes some pixels in each image block. For ease of description, some pixels in an image block are referred to as pixel blocks below.

[0094] For each image block, multiple pixel blocks can be obtained, and the number of pixels included in any two pixel blocks in the multiple pixel blocks can be the same or different. In addition, the pixels included in each pixel block can be adjacent pixels or non-adjacent pixels. Exemplarily, it can be understood with reference to formula 1-2.

[0095]

[0096] Wherein, i=1, 2, ..., n. n is a positive integer greater than 1, and is used to represent the number of image blocks after segmentation. m is a positive integer greater than 1, and is used to represent the number of pixel blocks included in an image block. c is used to represent the length of a vector corresponding to a pixel block.

[0097] If there are n image blocks, there are n pixel block groups, which can be understood by referring to formula 1-3.

[0098]

[0099] It should be noted that the pixels included in any two pixel blocks among the multiple pixel blocks may overlap.

[0100] For example, a method for obtaining an element set is given below. The element set can be obtained by image matrix column conversion (imageto column, im2col). im2col mainly converts the data contained in each window in the image data into a column vector, and finally arranges the data into a new matrix by column. Figure 5 To explain, Figure 5 As shown, each number represents a pixel. Figure 5 The channel dimension of each pixel is not shown in . The image is traversed by sliding a window, where the size of the window can be customized, such as Figure 5 The window is 3×3 in the image, and can also be customized to windows of other sizes, such as 2×2, 4×4, etc., which is not limited in the embodiments of the present application. The step size of each sliding of the window can also be customized, such as sliding a distance of 1 pixel each time, sliding a distance of two pixels each time, etc., which is not limited in the embodiments of the present application. Each time the window is slid on the image block, all the pixels included in the window can be regarded as a pixel block. Since each pixel block has a channel dimension, after each pixel is expanded, each pixel corresponds to multiple positions in a column vector. For example, each pixel can include three channels of red, green and blue. After each pixel block is expanded, it corresponds to 3 element positions in a column vector. Each pixel block can be converted into a row vector or a column vector, such as Figure 5 As shown in , the process of converting a pixel block into a column vector is shown.

[0101] 403. Perform feature extraction on the first vector using a first feature extraction model to obtain a first feature, and perform feature extraction on the second vector using a second feature extraction model to obtain a second feature.

[0102] The first vector is used to represent the segmented object to be processed, such as the first vector is used to represent the image block mentioned in step 401 and step 402. The second vector is used to represent some elements in the segmented object, such as the second vector is used to represent the pixel block mentioned in step 401 and step 402.

[0103] The first feature extraction model and the second feature extraction model can be understood as the multiple feature extraction modules in the first machine learning model / rule 201 mentioned above. For example, the first feature extraction model and the second feature extraction model can be CNN or RNN. The first feature extraction model includes multiple feature extraction modules, and the second feature extraction model includes multiple feature extraction modules. For one of the first feature extraction model or the second feature extraction model, multiple feature extraction modules are connected end to end, and the output of the previous feature extraction module is used as the input of the next feature extraction module, so that the next feature extraction module continues to extract features. Each feature extraction module has a specific weight matrix, and its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The input is traversed through the weight matrix to complete the work of extracting specific features from the image. For the feature extraction module of the current first feature extraction model, the output of the previous feature extraction module can be regarded as the first feature. Among them, image features mainly include color features, texture features, shape features, and spatial relationship features of the image. Color features are a global feature that describes the surface properties of the scene corresponding to the image or image area; generally, color features are based on pixel features, and at this time all pixels belonging to the image or image area have their own contributions. Since color is insensitive to changes in the direction, size, etc. of an image or image region, color features cannot capture the local features of an object in an image very well. Texture features are also a global feature, which also describes the surface properties of the scene corresponding to the image or image region; but since texture is only a characteristic of the surface of an object and cannot fully reflect the essential properties of the object, it is impossible to obtain high-level image content using only texture features. Unlike color features, texture features are not based on pixel features, and they need to be statistically calculated in an area containing multiple pixels. There are two types of representation methods for shape features, one is contour features and the other is regional features. The contour features of an image are mainly aimed at the outer boundaries of an object, while the regional features of an image are related to the entire shape region; spatial relationship features refer to the spatial position or relative direction relationship between multiple targets segmented from an image. These relationships can also be divided into connection / adjacency relationships, overlap / overlapping relationships, and inclusion / inclusion relationships. Generally, spatial position information can be divided into two categories: relative spatial position information and absolute spatial position information. The former relationship emphasizes the relative situation between targets, such as up, down, left, and right relationships, while the latter relationship emphasizes the distance, size, and orientation between targets. It should be noted that the above-listed image features can be used as some examples of specific features in an image. An image may also have other features, such as higher-level features: semantic features, which will not be expanded here.

[0104] 404. Fuse at least two second features according to the first target weight to obtain a first fused feature.

[0105] For a feature extraction module in the second feature extraction model, the feature extraction module fuses at least two second features according to the first target weight to obtain a first fused feature. The first fused feature is obtained to establish a connection between pixel blocks. Among them, establishing an association relationship between pixel blocks can be understood as taking into account the influence of one or more other pixel blocks on the pixel block when extracting the image features of a pixel block. The greater the influence of one or more other pixel blocks on the pixel block, the greater the weight, and the smaller the influence of one or more other pixel blocks on the pixel block, the smaller the weight. The influence of one or more pixel blocks on the pixel block can be measured in different ways, for example, it can be measured by the similarity between the vectors corresponding to the two pixel blocks, and specifically, it can be measured by the size of the inner product between the vectors corresponding to the two pixel blocks. The larger the inner product between the vectors corresponding to the two pixel blocks, the higher the similarity between the two pixel blocks, and the larger the weight. For another example, the similarity between pixel blocks can be obtained by training a neural network model. The vectors corresponding to the two pixel blocks can also be subjected to preset operations, and the influence of other pixel blocks on the pixel block can be obtained according to the results of the preset operations. For example, the vector corresponding to the pixel block to be processed and the vector corresponding to the pixel block adjacent to the pixel block to be processed can be averaged, and the average value can be superimposed on the vector corresponding to the pixel block to be processed.

[0106] In a preferred embodiment, the second feature extraction model may be a neural network based on a self-attention mechanism, for example, the second feature extraction model may be a Transformer. When the second feature extraction model is a Transformer, after the second vector is subjected to feature extraction by the second feature extraction model, the first fused feature may be obtained. The following takes the case where the second feature extraction model is a Transformer and the object to be processed is image data as an example to illustrate how the second vector is subjected to feature extraction by the second feature extraction model to obtain the second feature or the first fused feature. Figure 7 When the second feature extraction model is a Transformer, the multiple feature extraction modules in the second feature extraction model can be multiple feature extraction blocks for feature processing. The multiple blocks are connected end to end, and the output of the previous block is used as the input of the next block, so that the next block continues to extract features. Each block has a specific weight matrix, and its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The weight matrix traverses the input to complete the work of extracting specific features from the image.

[0107] In the process of extracting image features, each block can establish associations between pixels (or pixel blocks) and pixels (or pixel blocks) in a variety of ways. When extracting features from pixel blocks, each block of the second feature extraction model performs self-attention calculations on multiple pixel blocks, taking into account the impact of each pixel block on the currently processed pixel block. Figure 7 , is a schematic diagram of the Transformer architecture. A block generally includes a standardization processing module, which is used to standardize the input, wherein the standardization processing can be understood as making the mean of the input data 0 and the standard deviation 1, so that the loss value decreases smoothly during each training process. The output of the standardization processing module can be regarded as the second feature. A block can also include a self-attention module, and the output of the standardization processing module is used as the input of the self-attention module. When a block is a block of the second feature extraction model, the self-attention module performs self-attention calculation on multiple pixel blocks to establish an association relationship between pixel blocks. A block can also include another standardization processing module, which standardizes the output of the self-attention module so that the loss value can decrease more smoothly during each training process. For the current block, the output of the previous block of the current block can be regarded as the second feature, the input of the previous block can be regarded as the second vector, the output of the previous block is used as the input of the current block, and the output of the current block is the first fusion feature. For the next block of the current block, the output of the current block can be regarded as the second feature, and the output of the next block is the first fusion feature. It should be noted that when the current block is the first block, the input of the current block is not the second feature but can only be the second vector.

[0108] In a preferred embodiment, the first feature extraction model and the second feature extraction model are the models mentioned above that require the input to be a one-dimensional vector. For example, the first feature extraction model can be one of Transformer, GRU and LSTM, and the second feature extraction model can be one of Transformer, GRU and LSTM. As mentioned above, the input requirement of these feature extraction models is a one-dimensional vector. Therefore, for these models, after obtaining the image block and the pixel block through steps 401 and 402, it is necessary to convert the image block into a vector representation, and use the image block represented by the vector as the input of the first feature extraction model, and convert the pixel block into a vector representation, and use the pixel block represented by the vector as the input of the second feature extraction model. Among them, there are many ways to convert the image block into a vector representation. For example, refer to Figure 6To understand, the pixels of each row of the image block can be spliced ​​head to tail. Since each pixel block has a channel dimension, after each pixel is expanded, each pixel corresponds to multiple positions in a column vector. For example, each pixel can include three channels of red, green and blue. After each pixel block is expanded, it corresponds to 3 element positions in a column vector. Each image block can be converted into a row vector or a column vector. After the vectors corresponding to all image blocks are sorted by row or column, the vector matrix of the image block can be obtained. The method of converting pixel blocks into vector representation can be understood by referring to the method of converting image blocks into vector representation. For example, in the above example of obtaining pixel blocks through im2col, all pixels included in each window are expanded into a column vector, and multiple column vectors can be obtained, and then sorted by column to obtain the vector matrix of the pixel block. In addition, the first feature extraction model and the second feature extraction model may also have requirements for the size of the input vector, such as only vectors of a preset length can be used as inputs of the first feature extraction model and the second feature extraction model. Therefore, in some possible implementations, it is also necessary to perform mapping processing on the transformed vector of each image block so as to map the transformed vector of the image block into a vector of a preset length to meet the input requirements of the first feature extraction model; it is also necessary to perform mapping processing on the transformed vector of each pixel block so as to map the transformed vector of the pixel block into a vector of a preset length to meet the input requirements of the second feature extraction model. The first vector in the present application is used to represent the object to be processed after segmentation, such as the first vector is used to represent the image block mentioned in step 401 and step 402. The second vector is used to represent some elements in the segmented object, such as the second vector is used to represent the pixel block mentioned in step 401 and step 402. If the first vector is used as the input of the first feature extraction model and the second vector is used as the input of the second feature extraction model, the first vector also meets the input requirements of the first feature extraction model and the second vector also meets the input requirements of the second feature extraction model.

[0109] 405. Fuse the first feature and the first fused feature to obtain a second fused feature, where the second fused feature is used to obtain a feature of the object to be processed.

[0110] For a feature extraction module in the second feature extraction model, the solution provided in the present application can realize the fusion processing of the first feature and the first fusion feature in a variety of ways. The following will respectively explain from two aspects of fusion timing and fusion method.

[0111] First, let me explain the fusion timing: Figure 8In sub-graph a in the figure, the solution provided by the present application enables the first machine learning model / rule 201 to include two feature extraction models, namely, a first feature extraction model and a second feature extraction model. Among them, the second feature extraction model establishes an association relationship between pixels (or pixel blocks) and pixels (or pixel blocks), which can be understood specifically with reference to step 403. The first feature extraction model can establish an association relationship between image blocks and image blocks. Regarding how to establish an association relationship between image blocks and image blocks, you can refer to how to establish an association relationship between pixel blocks and pixel blocks for understanding. Specifically, when extracting the image features of an image block, the influence of one or more other image blocks on the image block is taken into account, which will not be repeated here. It is introduced above that the first feature extraction model includes multiple feature extraction modules, among which, for the current feature extraction module, the output of the previous feature extraction module is used to obtain the input of the next feature extraction module, and the input of the previous feature extraction module can be regarded as the first vector. Continue to refer to Figure 8 In sub-figure a in FIG. 1 , in a possible implementation, the first feature output by the previous feature extraction module in the first feature extraction model and the first fused feature output by the current feature extraction module in the second feature extraction model can be fused to obtain a second fused feature, and the second fused feature can be used as the input of the current feature extraction module in the first feature extraction model. Figure 8 In sub-figure b in the figure, in a possible implementation, the first feature output by the previous feature extraction module in the first feature extraction model can be used as the input of the current feature extraction module in the first feature extraction model, and the output of the current feature extraction module in the first feature extraction model and the first fused feature output by the current feature extraction module in the second feature extraction model are fused to obtain the second fused feature as the input of the next feature extraction module in the first feature extraction model.

[0112] The fusion method is further explained: In one possible implementation, when the first fused feature includes multiple features, the multiple first fused features can be spliced ​​end to end to obtain a spliced ​​feature. The spliced ​​feature is mapped to a feature of a target length, and the target length is determined according to the length of the first feature. If the length of the spliced ​​feature is the same as that of the first feature, the two can be directly added, and the first feature and the feature of the target length are added to obtain a second fused feature. In one possible implementation, the first feature and the first fused feature are spliced ​​end to end to obtain a second fused feature, for example, the first feature and the spliced ​​feature are spliced ​​to obtain a second fused feature. In one possible implementation, a target operation is performed on the first feature and the first fused feature to obtain a second fused feature, and the target operation includes at least one of addition or multiplication.

[0113] The second fused features are used to determine the final features of the object to be processed. In one possible implementation, the second fused features output by the last feature extraction module among the multiple feature extraction modules in the first feature extraction model are used to determine the final extracted features of the object to be processed. For each image block, the last feature extraction module will output the corresponding second fused features, and the set of second fused features is the final feature of the object to be processed. In one possible implementation, the second fused features corresponding to each image block output by the last feature extraction module are weighted, and the result after weighted processing is used as the final feature of the object to be processed.

[0114] Depend on Figure 4 It can be seen from the corresponding embodiments that the solution provided by the present application retains the association relationship between image blocks and image blocks, and also retains the association relationship between pixels (or pixel blocks) and pixels (or pixel blocks) in the process of extracting image features through the first machine learning model / rule 201, so that the image features extracted by the first machine learning model / rule 201 can well capture the color features, texture features, shape features and spatial relationship features of the image, thereby improving the performance of the first machine learning model / rule 201. In addition, it should be noted that Figure 4 The corresponding embodiments are mainly described using image data as an example, but it should be clear that the solutions provided in this application are also applicable to other types of data. For example, for text data, in the process of extracting text features through the first machine learning model / rule 201, the association relationship between text blocks and text blocks is retained, and the association relationship between word blocks and word blocks is retained, so that the text features extracted by the first machine learning model / rule 201 can well capture the semantic features of the text, etc. The text block can be understood as including some elements in the text data to be processed, such as including some adjacent words in the text data to be processed. The word block can be understood as including some elements in the text block, such as including some adjacent words in the text block. The following embodiments are all described using image data as an example. Other types of data can be understood by referring to the processing flow of image data, and this will not be repeated below.

[0115] In order to enable the image features extracted by the first machine learning model / rule 201 to well capture the color features, texture features, shape features, spatial relationship features, etc. of the image, the position information of the image blocks and pixel blocks can also be retained in the process of the model extracting image features, which is explained below in conjunction with an embodiment.

[0116] See also Fig. 9 , which is a flow chart of a feature extraction method provided in an embodiment of the present application.

[0117] like Fig. 9As shown, a feature extraction method provided in an embodiment of the present application may include the following steps:

[0118] 901. Split the object to be processed to obtain the split object to be processed.

[0119] 902. For each segmented object to be processed, obtain multiple element sets.

[0120] Step 901 and step 902 can refer to Figure 4 The corresponding steps 401 and 402 in the embodiment are understood and will not be repeated here.

[0121] 903. Fuse the first position information on the first vector, and fuse the second position information on the second vector.

[0122] The first position information is the position information of the segmented object in the object to be processed. For example, the first position information is the position information of the image block in the image block, and the second position information is the position information of some elements in the segmented object in the segmented object, for example, the second position information is the position information of the pixel block in the image block.

[0123] Among them, the first position information can be represented by the coordinate information of a pixel or can be represented by the coordinate information of multiple pixels. For example, when the image to be processed is evenly divided to obtain multiple image blocks, the position information of each image block can be represented by the coordinates of the pixel in the upper left corner of each image block. For another example, when each image block is a regular rectangle or square, the position information of each image block can be represented by the coordinates of the pixel in the upper left corner and the coordinates of the pixel in the lower right corner of each image block. It should be noted that the coordinates of the pixel in the upper left corner and the coordinates of the pixel in the lower right corner here are only for example purposes, used to illustrate that the first position information can be represented by the coordinate information of a pixel or the coordinate information of multiple pixels, and does not represent a limitation of the solution provided in the present application.

[0124] The second position information can be represented by the coordinate information of one pixel or by the coordinate information of multiple pixels. In addition, since the pixels included in a pixel block can be non-adjacent pixels, the position information in a pixel block can be represented by the coordinates of all pixels included in the pixel block.

[0125] In addition to representing the first position information and the second position information by the coordinate information of the pixel, the first position information and the second position information can also be represented by a coding vector. Taking the first position information as an example, the first machine learning model / rule 201 can include a position coding module. In the initial state, the position coding module can randomly set a vector to represent the position information of each image block. In the process of iterative training of the first machine learning model / rule 201, the parameters of the position coding module can be updated according to the loss value, so that the vector encoded by the position coding module for representing the image block position information can be closer to the real position information of the image block.

[0126] The first position information is fused on the first vector, and the second position information is fused on the second vector. It can be understood as the X in formula 1-1. n , Y in formula 1-3 0 i It has been updated and can be understood by referring to Formula 1-4 and Formula 1-5.

[0127] X n ←X n +E position-patch (1-4)

[0128] Y 0 i ←Y 0 i +E position-pixel (1-5)

[0129] Among them, E position-patch Used to indicate the first position information, E position-pixel Used to indicate the second location information.

[0130] Figure 4 The corresponding embodiment introduces that in a possible implementation manner, the weight matrices of each feature extraction module in the first feature extraction model are all 0. Then the first vector only carries the first position information. For example: if the weight matrices of each extraction module in the first feature extraction model are all 0, then for the first feature extraction module in the first feature model, the input of the first feature extraction module is n first vectors, and the values ​​of all elements in each of the n first vectors are all 0. The n first vectors are fused with the first position information and serve as the input of the first feature extraction module. In a possible implementation manner, the input of the first feature extraction module can also be n+1 first vectors, and the values ​​of all elements in each of the n+1 first vectors are all 0. For the n+1 first vectors, n vectors are used to fuse the first position information, and the remaining one vector is used to represent the weighted average of the first position information corresponding to each first image block.

[0131] 904. Perform feature extraction on the first vector integrated with the first position information through a first feature extraction model to obtain a first feature, and perform feature extraction on the second vector integrated with the second position information through a second feature extraction model to obtain a second feature.

[0132] 905. Fuse at least two second features according to the first target weight to obtain a first fused feature.

[0133] 906. Fuse the first feature and the first fused feature to obtain a second fused feature, where the second fused feature is used to obtain features of the object to be processed.

[0134] Steps 904 to 906 can refer to Figure 4 The steps 403 to 405 in the corresponding embodiment are understood as follows. The difference is that Fig. 9 The corresponding embodiment can provide more information to the first feature extraction model and the second feature extraction model, specifically providing the first position information and the second position information. The more information the first feature extraction model and the second feature extraction can obtain, the more helpful it is for the first feature extraction model and the second feature extraction to learn and better extract image features.

[0135] Figure 4 as well as Fig. 9 In the embodiment, it is introduced that the solution provided by the present application enables the first machine learning model / rule 201 to include a first feature extraction model and a second feature extraction model, so that in the process of extracting image features through the first machine learning model / rule 201, the association relationship between image blocks and image blocks is retained, and the association relationship between pixels (or pixel blocks) and pixels (or pixel blocks) is retained, so that the image features extracted by the first machine learning model / rule 201 can well capture the color features, texture features, shape features and spatial relationship features of the image, etc., thereby improving the performance of the first machine learning model / rule 201. In some possible implementations, the first machine learning model / rule 201 may also include a larger number of first feature extraction models and a larger number of second feature extraction models. This is explained below in conjunction with a specific embodiment.

[0136] See also Fig.10 , is a schematic diagram of the architecture of a model provided in an embodiment of the present application. The model may include multiple feature extraction models, such as Fig.10The feature extraction model 1, feature extraction model 2 and feature extraction model 3 shown in . Among them, for feature extraction model 1, feature extraction model 1 is equivalent to the first feature extraction model, and feature extraction model 2 and feature extraction model 3 are both equivalent to the second feature extraction model; for feature extraction model 2, feature extraction model 1 is equivalent to the first feature extraction model, and feature extraction model 2 is equivalent to the second feature extraction model; or feature extraction model 2 is equivalent to the first feature extraction model, and feature extraction model 3 is equivalent to the second feature extraction model; for feature extraction model 3, feature extraction model 1 and feature extraction model 2 are both equivalent to the first feature extraction model, and feature extraction model 3 is equivalent to the second feature extraction model. The specific execution process of the first feature extraction model and the second feature extraction model has been described in . Figure 4 and Fig. 9 The corresponding embodiments have been described in detail and will not be repeated here.

[0137] In one possible implementation, the image to be processed may be segmented multiple times. For example, in one segmentation, the image to be processed is segmented into 4 image blocks, which are preprocessed to meet the input requirements of feature extraction model 1 and then used as the input of feature extraction model 1; in another segmentation, the image to be processed is segmented into 16 image blocks, which are preprocessed to meet the input requirements of feature extraction model 2 and then used as the input of feature extraction model 2; in another segmentation, the image to be processed is segmented into 64 image blocks, which are preprocessed to meet the input requirements of feature extraction model 3 and then used as the input of feature extraction model 3. It should be noted that, in one possible implementation, multiple feature extraction models, such as feature extraction model 1, feature extraction model 2 and feature extraction model 3, may be executed simultaneously and in parallel.

[0138] Through the solution provided by the embodiment of the present application, the performance of the feature extraction model can be improved, so that the extracted image features can better represent the image information. The more image information that the image features can represent, the more conducive it is to improving the accuracy of the visual analysis task. The following is an introduction to the solution provided by the present application by taking the application of the solution provided by the present application to several typical visual analysis tasks as an example.

[0139] See also Fig.11When the solution provided in the present application is applied to the application scenario of autonomous driving, the mature first machine learning model / rule 201 can be deployed on the autonomous driving vehicle or on the cloud device. After the autonomous driving vehicle obtains the environmental image around the vehicle through the camera, the obtained image is input into the preprocessing module so that the preprocessing model segments the image to obtain image blocks and pixel blocks, and converts the obtained image blocks and pixel blocks into vectors that meet the input requirements of the first feature extraction model and the second feature extraction model. Among them, the preprocessing module can be regarded as a part of the mature first machine learning model / rule 201, or as a separate part. When the preprocessing module is a separate part, the preprocessing module can be deployed on the autonomous driving vehicle, and the mature first machine learning model / rule 201 can be deployed on the cloud device. The first machine learning model / rule 201 performs feature extraction on the environmental image around the vehicle acquired by the camera through the first feature extraction model and the second feature extraction model. During the feature extraction process, both the correlation between image blocks and the correlation between pixel blocks are retained, so that the extracted image features can better characterize the area where each object in the environment around the vehicle is located, which is beneficial for the semantic segmentation model in the first machine learning model / rule 201 to segment the environmental image around the vehicle acquired by the camera according to the extracted features, so as to segment the areas where different objects such as the road surface, roadbed, vehicles, pedestrians, etc. are located from the image, thereby keeping the vehicle driving in the correct area.

[0140] See also Fig.12 When the solution provided by the present application is applied in the field of intelligent monitoring, the mature first machine learning model / rule 201 can be deployed on an intelligent monitoring device or on a cloud device. Fig.12The images acquired by the cameras A, B, and C shown in the figure are input into the preprocessing module, so that the preprocessing module segments the images to obtain image blocks and pixel blocks, and converts the acquired image blocks and pixel blocks into vectors that meet the input requirements of the first feature extraction model and the second feature extraction model. Among them, the preprocessing module can be regarded as a part of the mature first machine learning model / rule 201, or as a separate part. When the preprocessing module is a separate part, the preprocessing module can be deployed on the monitoring device, and the mature first machine learning model / rule 201 can be deployed on the cloud device. The first machine learning model / rule 201 extracts features from the images acquired by the intelligent monitoring device through the first feature extraction model and the second feature extraction model. Since in the process of feature extraction, both the association relationship between image blocks and image blocks and the association relationship between pixel blocks are retained, the extracted image features can better characterize the features of objects appearing within the perception range of the intelligent monitoring device. For example, if pedestrian attributes are to be identified, the image features extracted by the solution provided in this application can better characterize the attributes of the pedestrian and the detailed features of the pedestrian, which is conducive to the attribute recognition model in the first machine learning model / rule 201 to recognize the attributes of the pedestrian in the image obtained by the intelligent monitoring device based on the extracted features, such as identifying the pedestrian's gender, age, hair color, clothing, outfit, etc. The results of the pedestrian attribute recognition can be displayed on the terminal device or stored in the server.

[0141] In order to more intuitively understand the beneficial effects of this solution, the following is a description of the beneficial effects of the embodiment of this application in combination with data. During testing, the first machine learning model / rule 201 is used to perform the image classification task. Fig.13 , which is a flow chart of performing an image classification task through a first machine learning model / rule 201. The first machine learning model / rule 201 includes a plurality of target feature extraction models, each of which includes a first feature extraction model and a second feature extraction model. Fig.13 In which, L represents a positive integer. The first machine learning model / rule 201 may also include an image preprocessing module, or the image preprocessing module may also be a module independent of the first machine learning model / rule 201. The image preprocessing module segments the image to obtain image blocks and pixel blocks, and converts the obtained image blocks and pixel blocks into vectors that meet the input requirements of the first feature extraction model and the second feature extraction model. In addition, in one possible implementation, the vectors corresponding to the multiple image blocks that meet the input requirements of the first feature extraction model may be weighted, and the result may also be used as the input of the first feature extraction model. The relevant processes executed by the image preprocessing module and the target feature extraction model can be referred to. Figure 4 as well as Fig. 9 The first machine learning model / rule 201 may also include a multi-layer perceptron head (MLP head), which is used to perform an image classification task according to the output of the last target feature extraction model to output a classification result, such as Fig.13 In the corresponding scenario, the output classification result is “house.” The configuration information of the first machine learning model / rule 201 used in the first test is shown in Table 1.

[0142] Table 1:

[0143]

[0144] Parameter one is used to indicate the number of feature extraction modules included in the feature extraction model, that is, the first feature extraction model includes 12 feature extraction modules, and the second feature extraction model includes 12 feature extraction modules. Parameter two is used to indicate the requirements of the second feature extraction model for the length of the input vector. Parameter 3 is used to indicate the number of heads (multi-head self-attention) in the self-attention module in the second feature extraction model. Parameter four is used to indicate the requirements of the first feature extraction model for the length of the input vector. Parameter five is used for the number of heads (multi-head self-attention) in the self-attention module in the first feature extraction model. Parameter six is ​​used to indicate the total number of parameters in the first machine learning model / rule 201 to which a feature method provided in the present application is applied, and the unit of the number of parameters is million. Parameter seven is used to indicate the number of floating point operations (FLOPs), and the unit is billion.

[0145] The test data set is the ImageNet data set, and the image classification test experiment is conducted on the ImageNet data set. The test results are as follows: Fig.14 For display. Fig.14As shown, compared with several existing image classification models, the solution provided by the present application retains the association between image blocks and image blocks, and also retains the association between pixels (or pixel blocks) and pixels (or pixel blocks) in the process of extracting image features through the first machine learning model / rule 201, so that the image features extracted by the first machine learning model / rule 201 can well capture the color features, texture features, shape features, and spatial relationship features of the image, thereby improving the classification accuracy of the first machine learning model / rule 201. Specifically, when the computational complexity of several existing models and the first machine learning model / rule 201 that applies the feature extraction method provided by the present application is the same, the first machine learning model / rule 201 that applies the feature extraction method provided by the present application has a higher accuracy rate in classifying images. In addition, the test results also show that compared with existing image classification models, the first machine learning model / rule 201 that applies the feature extraction method provided by the present application has less computational complexity. In other words, the first machine learning model / rule 201 that applies the feature extraction method provided by the present application is more efficient. Specifically, when several existing models and the first machine learning model / rule 201 that applies the feature extraction method provided by the present application have the same accuracy in image classification, the first machine learning model / rule 201 that applies the feature extraction method provided by the present application requires less computational complexity.

[0146] The above introduces a feature extraction method provided in an embodiment of the present application. Through the feature extraction method provided in the present application, the extracted features of the object to be processed can better characterize the object to be processed, thereby improving the performance of the model to which the feature extraction method is applied.

[0147] It is understandable that, in order to achieve the above functions, related devices for implementing the above scheme are also provided below. These related devices include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0148] Specific reference Fig.15 , Fig.15A schematic diagram of a structure of an electronic device provided in an embodiment of the present application. The electronic device may include a first acquisition module 1501, a second acquisition module 1502, a first fusion module 1503, a second fusion module 1504 and a third fusion module 1505.

[0149] The first acquisition module 1501 is used to acquire a first feature, and the second acquisition module 1502 is used to acquire multiple second features. The first feature is acquired after the first feature extraction model performs feature extraction on the first vector. The first vector is used to represent the object after the first segmentation. The object after the first segmentation includes some elements in the object to be processed. The second feature is acquired after the second feature extraction model performs feature extraction on the second vector. The second vector is used to represent some elements in the object after the first segmentation. The first fusion module 1503 is used to fuse at least two second features according to the first target weight to acquire a first fused feature. The first target weight is determined according to the influence of each second feature of the at least two second features on the target second feature. The target second feature is any second feature of the at least two second features. The second fusion module 1504 is used to fuse the first feature and the first fused feature to acquire a second fused feature. The second fused feature is used to acquire the feature of the object to be processed.

[0150] In a possible implementation, the first acquisition module 1501 is also used to acquire a third feature, where the third feature is acquired after feature extraction of a third vector by a first feature extraction model, and the third vector is used to represent the object after the second segmentation, and the object after the second segmentation includes some elements in the object to be processed. The third fusion module 1505 is used to fuse the first feature and the third feature according to a second target weight to acquire a third fused feature, where the second target weight is determined according to the influence of the third feature on the first feature. The second fusion module 1504 is specifically used to fuse the third fused feature with the first fused feature to acquire a second fused feature.

[0151] In a possible implementation, the first vector is specifically used to represent the first segmented object carrying the first position information, where the first position information is position information of the first segmented object in the object to be processed.

[0152] In a possible implementation, each second vector is specifically used to represent some elements in the first segmented object that carry second position information, where the second position information is position information of some elements in the first segmented object in the first segmented object.

[0153] In a possible implementation, the second fusion module 1504 is specifically configured to perform end-to-end concatenation processing on the first feature and the first fused feature to obtain the second fused feature.

[0154] In a possible implementation, the second fusion module 1504 is specifically configured to perform a target operation on the first feature and the first fused feature to obtain a second fused feature, where the target operation includes at least one of addition or multiplication.

[0155] In a possible implementation, the second fusion module 1504 is specifically used for, when the first fusion feature includes multiple features, performing head-to-tail splicing processing on the multiple first fusion features to obtain a spliced ​​feature. The spliced ​​feature is mapped to a feature of a target length, and the target length is determined according to the length of the first feature. The first feature and the feature of the target length are added to obtain a second fusion feature.

[0156] In one possible implementation, the first fusion module 1503 is specifically used to take at least two second features as input of a target model, the output of the target model is a first fused feature, the target model includes one of a self-attention network Transformer, a convolutional neural network CNN, or a recurrent neural network RNN, when the target model is a Transformer, the first target weight is determined based on the inner product between each second feature of the at least two second features and the target second feature, when the target model is one of CNN or RNN, the first target weight is preset.

[0157] In a possible implementation, the object to be processed is an image to be processed, the first vector is specifically used to represent the image after the first segmentation, the image after the first segmentation specifically includes some pixels in the image to be processed, the second vector is specifically used to represent some pixels in the image after the first segmentation, and the second fusion feature is specifically used to obtain features of the image to be processed.

[0158] In one possible implementation, the electronic device may be Figure 2 The training device 220 described in or Figure 2 The execution device 210 described in .

[0159] It should be noted that Fig.15 The information interaction, execution process, etc. between the modules in the electronic merger shown in the application are the same as those in the application. Figures 4 to 10 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0160] The present application also provides an electronic device. Fig.16 , Fig.16 The electronic device 1400 may be provided with Figures 4 to 10The first machine learning model / rule 201 described in the embodiment of the present invention comprises a first feature extraction model and a second feature extraction model for performing Figures 4 to 10 . Specifically, the electronic device 1400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1422 (for example, one or more processors) and memory 1432, and one or more storage media 1430 (for example, one or more mass storage devices) storing application programs 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 may be temporary storage or permanent storage. In one embodiment, the memory 1432 is a random access memory (RAM), which can directly exchange data with the central processor 1422, and is used to load data 1444 and application programs 1442 and / or operating system 1441 for direct operation and use by the central processor 1422, usually as a temporary data storage medium for the operating system or other running programs. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the electronic device. Furthermore, the central processor 1422 can be configured to communicate with the storage medium 1430 and execute a series of instruction operations in the storage medium 1430 on the electronic device 1400 .

[0161] The electronic device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0162] It should be noted that the CPU 1422 is also used to execute Figures 4 to 10 The other steps performed by the first machine learning model / rule 201 are performed by the CPU 1422 Figures 4 to 10 The specific implementation methods and beneficial effects of the steps executed by the first machine learning model / rule 201 in the corresponding embodiment can be referred to Figures 4 to 10 The descriptions in the corresponding method embodiments will not be repeated here one by one.

[0163] The present application also provides an electronic device. Fig.17 , Fig.17The electronic device 1700 may be provided with Figures 4 to 10 The first machine learning model / rule 201 described in the embodiment of the present invention comprises a first feature extraction model and a second feature extraction model for performing Figures 4 to 10 Specifically, the electronic device 1700 includes: a receiver 1701, a transmitter 1702, a processor 1703 and a memory 1704 (wherein the number of the processor 1703 in the electronic device 1700 may be one or more, Fig.17 In the example of FIG. 1701 , a processor 1703 is used, where the processor 1703 may include an application processor 17031 and a communication processor 17032. In some embodiments of the present application, the receiver 1701, the transmitter 1702, the processor 1703 and the memory 1704 may be connected via a bus or other means.

[0164] The memory 1704 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1703. A portion of the memory 1704 may also include a non-volatile random access memory (NVRAM). The memory 1704 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0165] The processor 1703 controls the operation of the electronic device. In a specific application, the various components of the electronic device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0166] The method disclosed in the above embodiment of the present application can be applied to the processor 1703, or implemented by the processor 1703. The processor 1703 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1703. The above processor 1703 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1703 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1704, and the processor 1703 reads the information in the memory 1704 and completes the steps of the above method in combination with its hardware.

[0167] The receiver 1701 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1702 can be used to output digital or character information through the interface; the transmitter 1702 can also be used to send instructions to the disk group through the above interface to modify the data in the disk group; the transmitter 1702 can also include a display device such as a display screen.

[0168] In one embodiment of the present application, the application processor 17031 is used to execute Figures 4 to 10 The method for executing the first machine learning model / rule 201 described in the corresponding embodiment.

[0169] For application processor 17031 execution Figures 4 to 10 For the specific implementation of the function of the first machine learning model / rule 201 in the corresponding embodiment and the beneficial effects brought about, please refer to Figures 4 to 10 The descriptions in the corresponding method embodiments will not be repeated here one by one.

[0170] It should be understood that the above is only one example provided in the embodiment of the present application, and the vehicle may have more or fewer components than those shown, may combine two or more components, or may have different configurations of components.

[0171] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0172] The execution device and training device provided in the embodiment of the present application may be specifically a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit to enable the chip to execute the above Figures 4 to 10 The method for extracting features of the model described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0173] For details, please refer to Fig.18 , Fig.18 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 160, NPU 160 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1603, which is controlled by the controller 1604 to extract matrix data from the memory and perform multiplication operations.

[0174] In some implementations, the operation circuit 1603 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1603 is a two-dimensional systolic array. The operation circuit 1603 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1603 is a general-purpose matrix processor.

[0175] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1602 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1601 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1608.

[0176] The unified memory 1606 is used to store input data and output data. The weight data is directly transferred to the weight memory 1602 through the direct memory access controller (DMAC) 1605. The input data is also transferred to the unified memory 1606 through the DMAC.

[0177] The bus interface unit 1610 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1609 to obtain instructions from the external memory, and is also used for the storage unit access controller 1605 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0178] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1606 or to transfer weight data to the weight memory 1602 or to transfer input data to the input memory 1601.

[0179] The vector calculation unit 1607 includes multiple operation processing units, and further processes the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0180] In some implementations, the vector calculation unit 1607 can store the processed output vector to the unified memory 1606. For example, the vector calculation unit 1607 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1603, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1607 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1603, for example, for use in a subsequent layer in a neural network.

[0181] The controller 1604 is connected to an instruction fetch buffer 1609, which is used to store instructions used by the controller 1604; the unified memory 1606, the input memory 1601, the weight memory 1602 and the instruction fetch buffer 1609 are all on-chip memories. The external memory is private to the NPU hardware architecture.

[0182] Among them, the operations of each layer in the recurrent neural network can be performed by the operation circuit 1603 or the vector calculation unit 1607.

[0183] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0184] The present application also provides a chip, which includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit. The processing unit may execute computer-executable instructions stored in the storage unit to enable the chip to execute the above Figures 4 to 10 The method described in . Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc. Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a network processor (neural-network processing unit, NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0185] In an embodiment of the present application, a computer-readable storage medium is also provided, in which a program for training a model is stored, and when the program is run on a computer, the computer executes the above Figures 4 to 10 The method described in .

[0186] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the above Figures 4 to 10 The steps in the method described in the illustrated embodiment. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0187] The present application also provides a circuit system, which includes a processing circuit, wherein the processing circuit is configured to perform the above Figures 4 to 10 The illustrated embodiments describe steps in a method.

[0188] Through the description of the above implementation methods, the technical personnel in the relevant field can clearly understand that the present application can be implemented by pure software or software plus necessary general hardware, and of course it can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CLUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or dedicated circuits. However, for this application, software program implementation is a better implementation method in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including several instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application. In addition, the computer software product can also be embodied in the form of a control, a driver, an independent or downloadable software object, etc.

[0189] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in a sequence other than the content illustrated or described here. The term "and / or" in the present application is only a kind of association relationship describing the associated objects, indicating that there can be three kinds of relationships, for example, A and / or B can be represented: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are an "or" relationship. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some ports, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules can be selected according to actual needs to achieve the purpose of the present application solution.

Claims

1. A method for feature extraction, characterized in that: include: Acquire a first feature and multiple second features, wherein the first feature is obtained by performing feature extraction on a first vector through a first feature extraction model, the first vector is used to represent the first segmented object, the first segmented object includes some elements in the object to be processed, and the second feature is obtained by performing feature extraction on a second vector through a second feature extraction model, the second vector is used to represent some elements in the first segmented object; fusing at least two of the second features according to a first target weight to obtain a first fused feature, wherein the first target weight is determined according to a first parameter value, the first parameter value is used to represent a similarity between each of the at least two second features and a target second feature, the target second feature is any one of the at least two second features, or the first target weight is a second parameter value, and the second parameter value includes at least one preset constant; The first feature and the first fused feature are fused to obtain a second fused feature, where the second fused feature is used to obtain the feature of the object to be processed.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a third feature, where the third feature is obtained by performing feature extraction on a third vector using the first feature extraction model, and the third vector is used to represent the second segmented object, where the second segmented object includes some elements in the object to be processed; The fusing the first feature and the first fused feature to obtain a second fused feature includes: fusing the first feature and the third feature according to a second target weight to obtain a third fused feature, wherein the second target weight is determined according to a third parameter value, and the third parameter value is used to represent a similarity between the third feature and the first feature, or the second target weight is a fourth parameter value, and the fourth parameter value includes at least one preset constant; The third fusion feature and the first fusion feature are fused to obtain the second fusion feature.

3. The method according to claim 1 or 2, characterized in that: The first vector is specifically used to represent the first segmented object carrying first position information, where the first position information is position information of the first segmented object in the object to be processed.

4. The method according to claim 1 or 2, characterized in that: Each of the second vectors is specifically used to represent some elements in the first segmented object that carry second position information, and the second position information is position information of some elements in the first segmented object in the first segmented object.

5. The method according to claim 1 or 2, characterized in that: The fusing the first feature and the first fused feature to obtain a second fused feature includes: The first feature and the first fused feature are concatenated end to end to obtain the second fused feature.

6. The method according to claim 1 or 2, characterized in that: The fusing the first feature and the first fused feature to obtain a second fused feature includes: A target operation is performed on the first feature and the first fused feature to obtain the second fused feature, where the target operation includes at least one of addition or multiplication.

7. The method according to claim 6, characterized in that The performing a target operation on the first feature and the first fused feature to obtain the second fused feature includes: When the first fusion feature includes multiple features, performing head-to-tail splicing processing on the multiple first fusion features to obtain a spliced ​​feature; Mapping the concatenated features into features of a target length, where the target length is determined according to the length of the first features; The first feature and the feature of the target length are added to obtain the second fusion feature.

8. The method according to claim 1 or 2, characterized in that: The fusing at least two of the second features according to the first target weight to obtain a first fused feature includes: The at least two second features are used as inputs of a target model, the output of the target model is the first fused feature, the target model includes one of a self-attention network Transformer, a convolutional neural network CNN, or a recurrent neural network RNN, when the target model is the Transformer, the first target weight is determined based on the inner product between each of the at least two second features and the target second feature, when the target model is one of the CNN or the RNN, the first target weight is the second parameter value.

9. The method according to claim 1 or 2, characterized in that: The object to be processed is an image to be processed, the first vector is specifically used to represent the first segmented image, the first segmented image specifically includes some pixels in the image to be processed, the second vector is specifically used to represent some pixels in the first segmented image, and the second fusion feature is specifically used to obtain features of the image to be processed.

10. A feature extraction model, characterized in that: The feature extraction model includes a first feature extraction model and a second feature extraction model. The first feature extraction model is used to obtain a first feature, wherein the first feature is obtained by performing feature extraction on a first vector through the first feature extraction model, and the first vector is used to represent the object after the first segmentation, and the object after the first segmentation includes some elements in the object to be processed; The second feature extraction model is used to obtain a plurality of second features, where the second features are obtained by performing feature extraction on a second vector using the second feature extraction model, and the second vector is used to represent some elements in the object after the first segmentation; The second feature extraction model is further used to fuse at least two of the second features according to a first target weight to obtain a first fused feature, wherein the first target weight is determined according to a first parameter value, and the first parameter value is used to represent a similarity between each of the at least two second features and a target second feature, and the target second feature is any one of the at least two second features, or the first target weight is a second parameter value, and the second parameter value includes at least one preset constant; The first feature extraction model is further used to fuse the first feature and the first fused feature to obtain a second fused feature, and the second fused feature is used to obtain the feature of the object to be processed.

11. The feature extraction model according to claim 10, characterized in that: The first feature extraction model is further used for: Acquire a third feature, where the third feature is obtained by performing feature extraction on a third vector using the first feature extraction model, and the third vector is used to represent the second segmented object, where the second segmented object includes some elements in the object to be processed; The first feature extraction model is specifically used for: fusing the first feature and the third feature according to a second target weight to obtain a third fused feature, wherein the second target weight is determined according to a third parameter value, and the third parameter value is used to represent a similarity between the third feature and the first feature, or the second target weight is a fourth parameter value, and the fourth parameter value includes at least one preset constant; The third fusion feature and the first fusion feature are fused to obtain the second fusion feature.

12. The feature extraction model according to claim 10 or 11, characterized in that: The first vector is specifically used to represent the first segmented object carrying first position information, where the first position information is position information of the first segmented object in the object to be processed.

13. The feature extraction model according to claim 10 or 11, characterized in that: Each of the second vectors is specifically used to represent some elements in the first segmented object that carry second position information, and the second position information is position information of some elements in the first segmented object in the first segmented object.

14. The feature extraction model according to claim 10 or 11, characterized in that: The first feature extraction model is specifically used for: The first feature and the first fused feature are concatenated end to end to obtain the second fused feature.

15. The feature extraction model according to claim 10 or 11, characterized in that: The first feature extraction model is specifically used for: A target operation is performed on the first feature and the first fused feature to obtain the second fused feature, where the target operation includes at least one of addition or multiplication.

16. The feature extraction model according to claim 15, characterized in that: The first feature extraction model is specifically used for: When the first fusion feature includes multiple features, performing head-to-tail splicing processing on the multiple first fusion features to obtain a spliced ​​feature; Mapping the concatenated features into features of a target length, where the target length is determined according to the length of the first features; The first feature and the feature of the target length are added to obtain the second fusion feature.

17. The feature extraction model according to claim 10 or 11, characterized in that: The second feature extraction model is specifically used for: The at least two second features are used as inputs of a target model, the output of the target model is the first fused feature, the target model includes one of a self-attention network Transformer, a convolutional neural network CNN, or a recurrent neural network RNN, when the target model is the Transformer, the first target weight is determined based on the inner product between each of the at least two second features and the target second feature, when the target model is one of the CNN or the RNN, the first target weight is the second parameter value.

18. The feature extraction model according to claim 10 or 11, characterized in that: The object to be processed is an image to be processed, the first vector is specifically used to represent the first segmented image, the first segmented image specifically includes some pixels in the image to be processed, the second vector is specifically used to represent some pixels in the first segmented image, and the second fusion feature is specifically used to obtain features of the image to be processed.

19. An image processing method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a visual perception model to extract image features through a feature extraction model included in the visual perception model, wherein the feature extraction model is a feature extraction model described in any one of claims 10 to 18; Perform visual perception on the image to be processed according to the image features.

20. The image processing method according to claim 19, characterized in that: The visually perceiving the image to be processed according to the image features includes: The image to be processed is classified according to the image features to obtain a classification result of the image to be processed.

21. The image processing method according to claim 19, characterized in that: The step of obtaining the image to be processed comprises: Acquire an image to be processed through a sensor of the vehicle; The visually perceiving the image to be processed according to the image features includes: The image to be processed is semantically segmented according to the image features to obtain a region where a target object in the image to be processed is located, and the target object includes one or more of a person, a vehicle, and a road surface.

22. The image processing method according to claim 19, characterized in that: The step of obtaining the image to be processed comprises: Acquire the image to be processed through the monitoring device; The visually perceiving the image to be processed according to the image features includes: If it is identified according to the image features that the image to be processed includes a person, attributes of the person are identified according to the image features, and the attributes include one or more of gender, skin color, age, and clothing.

23. An electronic device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 9 is implemented.

24. A computer-readable storage medium, characterized in that: The invention comprises a program which, when being run on a computer, causes the computer to execute the method according to any one of claims 1 to 6.

25. A circuit system, characterized in that: The circuit system comprises a processing circuit configured to perform the method of any one of claims 1 to 9.

26. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are loaded and executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

27. A chip, characterized in that: The chip is coupled to a memory and is used to execute a program stored in the memory to perform the method according to any one of claims 1 to 9.

Images