Image recognition method, device, equipment and storage medium
By performing vectorization and encoding module fusion rule processing on images, the computational resource consumption in the image recognition process is reduced, solving the problem of wasted computational resources in deep self-attention transform networks and achieving more efficient image recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2023-06-28
- Publication Date
- 2026-05-12
AI Technical Summary
Existing image recognition methods are quite computationally expensive, especially when using deep self-attention transform networks, where significant computational resources are wasted.
By vectorizing the target image, multiple feature vectors are obtained, and then encoded in the self-attention network model according to the encoding module sequence and fusion rules. This reduces the number of feature vectors between adjacent encoding modules and gradually reduces the number of tokens through fusion rules, thereby reducing the computational load.
It significantly reduces the computational resource consumption in the image recognition process and accelerates the convergence of the self-attention network model without affecting the recognition effect.
Smart Images
Figure CN116843963B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically to computer vision, image processing, deep learning and other technical fields, and can be applied to smart city scenarios. In particular, it relates to an image recognition method, device, equipment and storage medium. Background Technology
[0002] The Deep Self-Attention Transformer (TET) framework was first proposed for natural language processing. TTE uses a self-attention mechanism to capture global contextual information. In image recognition, TTE-based encoding blocks can divide the input image into image patches, analogizing these patches to words (tokens) in a natural language processing task. Feature maps are generated using these tokens, and image recognition is performed based on these feature maps.
[0003] Currently, image recognition methods consume a lot of computing resources. Summary of the Invention
[0004] This disclosure provides an image recognition method, apparatus, device, and storage medium that can significantly reduce the computational resource consumption of the image recognition process.
[0005] According to a first aspect of this disclosure, an image recognition method is provided, the method comprising:
[0006] The target image is vectorized to obtain at least two first feature vectors. The first feature vectors are then encoded sequentially through each encoding module according to the sequence of encoding modules in the self-attention network model and a first fusion rule to obtain a second feature vector. The encoding modules include at least two modules, and the first fusion rule includes: at the target location between at least one set of two adjacent encoding modules, the feature vectors encoded by the preceding encoding module in the two adjacent encoding modules are fused, the number of fused feature vectors being less than the number of feature vectors encoded by the preceding encoding module; the fused feature vectors are then input into the following encoding module in the two adjacent encoding modules for further encoding; the recognition result of the target image is determined based on the second feature vectors.
[0007] According to a second aspect of this disclosure, an image recognition apparatus is provided, the apparatus comprising: a vectorization unit, an encoding unit, and a determination unit.
[0008] The vectorization unit is used to vectorize the target image to obtain at least two first feature vectors; the encoding unit is used to encode the first feature vectors sequentially through each encoding module according to the sequence of encoding modules in the self-attention network model and the first fusion rule to obtain a second feature vector; wherein, the encoding module includes at least two modules, and the first fusion rule includes: at the target position between at least one set of two adjacent encoding modules, the feature vectors encoded by the previous encoding module in the two adjacent encoding modules are fused, the number of fused feature vectors is less than the number of feature vectors encoded by the previous encoding module, and the fused feature vectors are input into the next encoding module in the two adjacent encoding modules for further encoding; the determination unit is used to determine the recognition result of the target image based on the second feature vector.
[0009] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.
[0010] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the method described in the first aspect.
[0011] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to the first aspect.
[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0014] Figure 1 A schematic flowchart of the image recognition method provided in this embodiment of the disclosure;
[0015] Figure 2 This is another schematic flowchart of the image recognition method provided in the embodiments of this disclosure;
[0016] Figure 3 is a schematic diagram of dividing the encoding module into encoding sets;
[0017] Figure 4 Provided for the embodiments of this disclosure Figure 1A schematic diagram of an implementation process for S103 in the middle;
[0018] Figure 5 This is a schematic diagram illustrating the composition of the image recognition device provided in the embodiments of this disclosure;
[0019] Figure 6 This is another schematic diagram of the image recognition device provided in the embodiments of this disclosure;
[0020] Figure 7 A schematic block diagram of an example electronic device 700 provided for implementation of embodiments of the present disclosure. Detailed Implementation
[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0022] It should be understood that in the embodiments of this disclosure, the character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.
[0023] The Deep Self-Attention Transformer (TET) framework was first proposed for natural language processing. TTE uses a self-attention mechanism to capture global contextual information. In image recognition, TTE-based encoding blocks can divide the input image into image patches, analogizing these patches to words (tokens) in a natural language processing task. Feature maps are generated using these tokens, and image recognition is performed based on these feature maps.
[0024] Currently, image recognition methods consume a lot of computing resources.
[0025] For example, the Transformer framework is a powerful feature extraction paradigm with outstanding performance in machine translation tasks. The vision transformer (ViT) is a pioneering work on applying Transformer to computer vision tasks. ViT applies the coded blocks from the standard Transformer to vision tasks, dividing the input image into image patches and analogizing these patches to tokens in natural language processing tasks. Common ViT implementations primarily use multi-head attention mechanisms to obtain attention relationships between different tokens. Multi-head attention is a key component of Transformer networks used to learn the relationships between different locations in the input sequence. It consists of multiple independent heads, each learning a different relationship. Given a set of representations of the input sequence, multi-head attention first splits it into multiple heads, then performs independent query, key-value computation, attention weighting, and normalization operations on each head, and finally concatenates the outputs of all heads to form the final output. In this way, each head learns different local relationships in the input sequence, thereby improving the network's performance. In the image processing domain, the input sequence is typically a feature map obtained through image convolution, with each head exploring different visual patterns in the image. However, this approach consumes a large number of computational resources due to the large number of tokens.
[0026] Against this background, this disclosure provides an image recognition method that can significantly reduce the computational resource consumption of the image recognition process.
[0027] For example, the entity executing this image recognition method can be a computer or server, or it can be other devices with data processing capabilities. No limitation is placed on the entity executing this method.
[0028] In some embodiments, the server can be a single server, or it can be a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. This disclosure does not limit the specific implementation of the server.
[0029] Figure 1 This is a schematic flowchart illustrating the image recognition method provided in an embodiment of this disclosure. Figure 1 As shown, the method may include S101-S104.
[0030] S101. Vectorize the target image to obtain at least two first feature vectors.
[0031] For example, the image to be recognized can be used as input, and the recognition result of the image can be used as output to train a self-attention neural network, resulting in a self-attention network model. The obtained self-attention network model has the function of outputting the recognition result of the image based on the input image. The target image can be input into the self-attention network model, and the image processing module in the self-attention network model can vectorize the target image to obtain at least two first feature vectors corresponding to the target image. In this scheme, image vectorization can also be called image tokenization. The at least two first feature vectors corresponding to the target image can also be understood as at least two tokens corresponding to the target image.
[0032] Based on the above embodiments, for example, if the resolution of the target image input to the self-attention network model is 1000×1000, then according to the actual scenario requirements, the target image can be vectorized, that is, the target image can be tokenized, and the target image can be cropped to a resolution of 14×14. At this time, the cropped image will have 196 tokens.
[0033] S102. For the first feature vector, according to the sequence of the encoding modules in the self-attention network model and the first fusion rule, it is encoded through each encoding module in turn to obtain the second feature vector.
[0034] The encoding module includes at least two modules. The first fusion rule includes: at the target position between at least one set of two adjacent encoding modules, the feature vectors encoded by the first encoding module in the two adjacent encoding modules are fused. The number of fused feature vectors is less than the number of feature vectors encoded by the first encoding module. The fused feature vectors are then input into the second encoding module in the two adjacent encoding modules for further encoding.
[0035] For example, each first feature vector obtained in S101 can be fed into a series of encoding modules in the vision transformer (ViT) of the attention network model. Each input first feature vector is encoded one by one through each encoding module in sequence. For instance, a first feature vector is input into the first encoding module for the first encoding, and the first encoding module outputs the first encoded feature vector. Then, the encoded feature vector enters the second encoding module, the second encoding module encodes the first encoded feature vector for the second time, and outputs the second encoded feature vector, and so on, until the first feature vector passes through each encoding module in the vision transformer. The feature vector output by the last encoding module in the vision transformer is the second feature vector.
[0036] During the encoding process, the first fusion rule can be followed to fuse the feature vectors output by the first encoding module after the first encoding between the first encoding module and the second encoding module. The number of fused feature vectors is less than the number of feature vectors after the first encoding. Then, the fused feature vectors are input into the second encoding module to continue encoding.
[0037] Based on the above embodiments, for example, assuming there are 196 first feature vectors, the visual converter of the self-attention network model has four encoding modules, namely encoding module 1, encoding module 2, encoding module 3, and encoding module 4. The 196 first feature vectors can be encoded sequentially through each encoding module. That is, the 196 first feature vectors are input into encoding module 1, which encodes the 196 first feature vectors and outputs 196 encoded feature vectors. Then, the 196 encoded feature vectors are fused to obtain 98 fused feature vectors. These 98 fused feature vectors are then input into encoding module 2, which encodes the 98 fused feature vectors and outputs 98 encoded feature vectors. These 98 encoded feature vectors are then fused to obtain 49 fused feature vectors, and so on. Encoding module 4 can output 28 encoded feature vectors.
[0038] Optionally, after each encoding module outputs the encoded feature vector, it can be selected whether to fuse the encoded feature vectors according to actual needs, ensuring that the feature vectors are fused at least once during the entire encoding process.
[0039] S103. Determine the recognition result of the target image based on the second feature vector.
[0040] For example, the second feature vector can be one or more. When there is only one second feature vector, a feature map corresponding to the target image can be generated based on the second feature vector using pooling. When there are multiple second feature vectors, they can be fused into one first, and then a feature map corresponding to the target image can be generated based on this fused feature vector using pooling. Then, a self-attention network model is used to identify the feature map corresponding to the target image, and the recognition result of the target image is obtained according to the specific function of the self-attention network model.
[0041] This disclosure vectorizes the target image to obtain at least two first feature vectors. Then, these first feature vectors are encoded sequentially through each encoding module according to the sequence of encoding modules in the self-attention network model and a first fusion rule. During encoding, at the target position between at least one pair of adjacent encoding modules, the feature vectors encoded by the preceding encoding module in the adjacent two modules are fused. The number of fused feature vectors is less than the number of feature vectors encoded by the preceding encoding module. The fused feature vectors are then input into the following encoding module in the adjacent two modules for further encoding, ultimately obtaining the second feature vector corresponding to the target image. Based on the second feature vector, the recognition result of the target image is determined. For tasks such as classification, as the model deepens, the similarity between subsequent tokens increases, and the similarity between tokens tends to become identical. Maintaining so many tokens at this point would lead to resource waste. This scheme reduces the number of tokens by fusing them during the encoding process, thereby reducing the computational load during encoding and significantly reducing the computational resource consumption of the image recognition process without affecting the self-attention network model.
[0042] The method disclosed herein can also accelerate the convergence of self-attention network models during training.
[0043] Figure 2 This is another schematic flowchart illustrating the image recognition method provided in an embodiment of this disclosure. Figure 2 As shown, the method may include S201-S203.
[0044] S201. According to the order of the encoding modules, the encoding modules are divided into at least two encoding sets, and each encoding set includes at least one encoding module.
[0045] For example, a self-attention network model may have many encoding modules arranged in a certain order. These encoding modules can be divided into at least two encoding sets according to their order, with each encoding set containing at least one encoding module. The number of encoding sets can represent the theoretical number of feature vector fusions that can be performed.
[0046] Based on the above embodiments, Figure 3 is, for example, a schematic diagram of dividing the encoding module into encoding sets. (Reference) Figure 3A The coding modules can be divided into coding sets by dividing the total number of coding modules equally. (See reference) Figure 3B Alternatively, the encoding module can be divided into encoding sets in other ways. The specific method for dividing the encoding module into encoding sets is not limited here.
[0047] S202. Determine at least one pair of adjacent target coding sets from the coding set.
[0048] For example, at least one pair of adjacent target encoding sets can be determined from the encoding sets divided in S201 to prepare for the next step of determining the position for fusing the first feature vector.
[0049] S203. In two adjacent target encoding sets, the last encoding module of the first target encoding set and the first encoding module of the second target encoding set are taken as the target position for fusing the first feature vector.
[0050] For example, the number of target locations can represent the actual number of feature vector fusion operations.
[0051] Based on the above embodiments, by way of example, refer to Figure 3A Encoding set 1 and encoding set 2 can be used as the target encoding set. At this time, the target position for fusing the first feature vector is between encoding module 3 and encoding module 4.
[0052] This embodiment divides the encoding module into at least two encoding sets according to the order of the encoding modules. From the encoding sets, at least one pair of adjacent target encoding sets are determined. The encoding module at the end of the first target encoding set and the encoding module at the beginning of the second target encoding set are used as the target position for fusing the first feature vector. This method divides the encoding module into encoding sets, avoiding the introduction of additional calculation steps that would consume a lot of computing resources while reducing the number of vectors, thus reducing resource waste.
[0053] In some embodiments, the above-mentioned fusion of the feature vectors encoded by the preceding encoding module in at least one set of two adjacent encoding modules may include: fusing the feature vectors encoded by the preceding encoding module in the two adjacent encoding modules at each target position according to a second fusion rule for the target position.
[0054] For example, after determining the target location, a second fusion rule can be determined for each target location. The second fusion rule for each target location is a specific rule for fusing feature vectors based on the actual scenario requirements, indicating that the feature vectors are fused at the target location according to the preset rule. After determining the fusion rule for each target location, the feature vectors are fused at each target location according to the corresponding fusion rule.
[0055] Based on the above embodiments, for example, suppose there are target positions 1, 2, and 3, the fusion rule corresponding to target position 1 is A, the fusion rule corresponding to target position 2 is B, and the fusion rule corresponding to target position 3 is C. Then, the feature vectors arriving at target position 1 are fused according to fusion rule A at target position 1, the feature vectors arriving at target position 2 are fused according to fusion rule B at target position 2, and the feature vectors arriving at target position 3 are fused according to fusion rule C at target position 3.
[0056] This embodiment fuses the first feature vector at each target location according to the second fusion rule of the target location, thereby further reducing resource consumption.
[0057] In some embodiments, the aforementioned target locations include at least two, and in two adjacent target locations, the width or height of the first feature vector of the second target location is half the width or height of the first feature vector of the first target location.
[0058] For example, the feature vectors need to be fused according to a certain fusion rule. This fusion rule can be set as follows: among two adjacent target positions, the width or height of the first feature vector of the second target position is half the width or height of the first feature vector of the first target position.
[0059] Based on the above embodiments, for example, the resolution of the first feature vector at the first target location is 14×14, which can be expressed as the width of the first feature vector at the first target location being 14 and the height being 14. If the fusion rule for the first target location is that the width of the first feature vector at the second target location is half the width of the first feature vector at the first target location, while the height remains unchanged, then after fusion at the first target location, the width of the feature vector reaching the second target location is 7 and the height is 14, which can be expressed as the resolution of the feature vector at the second target location being 7×14.
[0060] This embodiment limits the aforementioned target locations to at least two, and in two adjacent target locations, the width or height of the first feature vector of the second target location is half the width or height of the first feature vector of the first target location. This allows the determined fusion rule to achieve the effect of reducing computational load and lowering resource consumption.
[0061] In some embodiments, the target locations include at least two, where, in two adjacent target locations, the first target location is fused according to width and the second target location is fused according to height, or the first target location is fused according to height and the second target location is fused according to width.
[0062] For example, when determining the fusion rule for each target location, the feature vectors of two adjacent target locations can be fused sequentially from different dimensions. For instance, the first target location can be fused according to width, the second target location according to height, the third target location according to width, and the fourth target location according to height.
[0063] Based on the above embodiments, for example, assume that four target positions are determined, and these four target positions are arranged in sequence, i.e., target position 1 is adjacent to target position 2, target position 2 is adjacent to target position 3, and target position 3 is adjacent to target position 4. A 14×14 feature vector is transmitted to target position 1, and the vector is fused according to width to form a 7×14 feature vector. The 7×14 feature vector is transmitted to target position 2, and the vector is fused according to height to form a 7×7 feature vector. The 7×7 feature vector is transmitted to target position 3, and the vector is fused according to width to form a 4×7 feature vector. The 4×7 feature vector is transmitted to target position 4, and the vector is fused according to height to form a 4×4 feature vector.
[0064] This embodiment defines the target location as including at least two. In two adjacent target locations, the first target location is merged according to width, and the second target location is merged according to height; alternatively, the first target location is merged according to height, and the second target location is merged according to width. Different merging rules can be provided, increasing the richness of merging rules and improving the flexibility of setting merging rules.
[0065] Figure 4 Provided for the embodiments of this disclosure Figure 1 A schematic diagram of one implementation process of S103. For example... Figure 4 As shown, Figure 1 The S103 shown may include S401-S403.
[0066] S401. The second feature vector is fused, and the number of the fused second feature vectors is 1.
[0067] For example, multiple second feature vectors can be fused into one feature vector according to the corresponding fusion rules.
[0068] Based on the above embodiments, for example, assuming the second feature vector is represented as 4×4, it can be fused into 2×2 and then into 1×1 by fusing the width and height of the first feature vector of the second target position into half the width and height of the first feature vector of the first target position.
[0069] S402. Generate the feature map corresponding to the target image based on the fused second feature vector.
[0070] For example, the second feature vector includes at least two. Multiple second feature vectors can be fused into one first, and then the feature map corresponding to the target image can be generated by pooling based on the fused feature vector.
[0071] Based on the above embodiments, for example, assuming there are 29 second feature vectors, the 29 second feature vectors are first fused into 1 feature vector, and this 1 feature vector is pooled to obtain the feature map corresponding to the target image.
[0072] S403. Recognize the feature map to obtain the recognition result of the target image.
[0073] For example, the feature map generated by S402 can be identified by a self-attention network model, and the recognition result of the corresponding target image can be obtained according to the specific function of the self-attention network model.
[0074] This embodiment fuses the second feature vector, resulting in a single second feature vector. Then, based on this fused second feature vector, a feature map corresponding to the target image is generated. Finally, the feature map is used for recognition to obtain the recognition result of the target image. This reduces the computational complexity of obtaining the feature map corresponding to the target image from the second feature vector.
[0075] In some embodiments, S402 above may include: performing global average pooling on the second feature vector using a self-attention network model to obtain a feature map corresponding to the target image.
[0076] For example, the second feature vector can be globally averaged through the pooling layer of a self-attention network model to obtain the feature map corresponding to the target image.
[0077] Alternatively, the second feature vector can be pooled through the pooling layer of the self-attention network model using max pooling or trainable pooling. The specific method for pooling the second feature vector to obtain the feature map corresponding to the target image is not limited here.
[0078] This embodiment uses a self-attention network model to perform global average pooling on the second feature vector to obtain the feature map corresponding to the target image. This reduces the computational complexity of obtaining the feature map and improves the recognition efficiency of the self-attention network model.
[0079] In some embodiments, the self-attention network model includes any of the following: image classification model, image detection model, and image localization model.
[0080] For example, this solution can be applied to any of the image classification models, image detection models, and image localization models. For image classification tasks, the self-attention network model serves as the image classification model. After obtaining the feature map corresponding to the target image through the aforementioned steps, the feature map can be input into the classification layer of the image classification model, which then outputs the classification result. During image classification model training, loss calculations can be performed through the classification layer to assess model performance and optimize the model.
[0081] This embodiment defines the self-attention network model as any one of the following: image classification model, image detection model, and image localization model. This demonstrates that the self-attention network model can be used for different image processing tasks, helping users better understand and utilize its technical advantages and application value.
[0082] In an exemplary embodiment, this disclosure also provides an image recognition apparatus that can be used to implement the image recognition method as described in the foregoing embodiments.
[0083] Figure 5 This is a schematic diagram illustrating the composition of an image recognition device provided in an embodiment of this disclosure. Figure 5 As shown, the device may include: a vectorization unit 501, an encoding unit 502, and a determination unit 503.
[0084] The vectorization unit 501 is used to vectorize the target image to obtain at least two first feature vectors.
[0085] Encoding unit 502 is used to encode the first feature vector sequentially through each encoding module according to the sequence of encoding modules in the self-attention network model and the first fusion rule to obtain the second feature vector; wherein, the encoding module includes at least two, and the first fusion rule includes: at the target position between at least one set of two adjacent encoding modules, the feature vectors encoded by the previous encoding module in the two adjacent encoding modules are fused, the number of fused feature vectors is less than the number of feature vectors encoded by the previous encoding module, and the fused feature vectors are input into the next encoding module in the two adjacent encoding modules for further encoding.
[0086] The determining unit 503 is used to determine the recognition result of the target image based on the second feature vector.
[0087] Figure 6 This is another schematic diagram of an image recognition device provided in an embodiment of this disclosure. Optionally, as shown... Figure 6 As shown, the device may include: a vectorization unit 601, an encoding unit 602, a determination unit 603, and a division unit 604.
[0088] The vectorization unit 601 can refer to the vectorization unit 501 in the foregoing embodiments.
[0089] The encoding unit 602 can refer to the encoding unit 502 in the foregoing embodiments.
[0090] The determining unit 603 can be referred to as the determining unit 503 in the foregoing embodiments.
[0091] The partitioning unit 604 is used to divide the encoding module into at least two encoding sets according to the order of the encoding modules, and each encoding set includes at least one encoding module.
[0092] The determining unit 603 is also used to determine at least one pair of adjacent target coding sets from the coding set;
[0093] The determining unit 603 is also used to determine the target position for fusing the first feature vector between the last encoding module of the first target encoding set and the first encoding module of the second target encoding set in two adjacent target encoding sets.
[0094] Optionally, the encoding unit 502 is specifically used to fuse the feature vectors encoded by the first encoding module in two adjacent encoding modules at each target location, according to the second fusion rule of the target location.
[0095] Optionally, the target locations include at least two, and in two adjacent target locations, the width or height of the first feature vector of the second target location is half the width or height of the first feature vector of the first target location.
[0096] Optionally, the target location includes at least two, and in two adjacent target locations, the first target location is merged according to width and the second target location is merged according to height, or the first target location is merged according to height and the second target location is merged according to width.
[0097] Optionally, the second feature vector includes at least two, and the determining unit 503 is specifically used to fuse the second feature vectors, the number of the fused second feature vectors being 1; based on the fused second feature vectors, a feature map corresponding to the target image is generated; the feature map is recognized to obtain the recognition result of the target image.
[0098] Optionally, the determining unit 503 is specifically used to perform global average pooling on the second feature vector through a self-attention network model to obtain the feature map corresponding to the target image.
[0099] Optionally, the self-attention network model includes any of the following: image classification model, image detection model, and image localization model.
[0100] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0101] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0102] In an exemplary embodiment, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in the above embodiments.
[0103] In an exemplary embodiment, the readable storage medium may be a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the above embodiments.
[0104] In an exemplary embodiment, the computer program product includes a computer program that, when executed by a processor, implements the method described in the above embodiments.
[0105] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0106] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0107] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0108] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as image recognition methods. For example, in some embodiments, the image recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the image recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform image recognition methods by any other suitable means (e.g., by means of firmware).
[0109] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0110] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0111] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0114] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0115] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. An image recognition method, the method comprising: Vectorize the target image to obtain at least two first feature vectors; The first feature vector is encoded sequentially through each of the encoding modules in the self-attention network model according to the sequence of the encoding modules and the first fusion rule to obtain the second feature vector; The encoding module includes at least two modules, and the first fusion rule includes: at the target position between at least one set of two adjacent encoding modules, fusing the feature vectors encoded by the first encoding module in the two adjacent encoding modules, wherein the number of fused feature vectors is less than the number of feature vectors encoded by the first encoding module, and inputting the fused feature vectors into the second encoding module in the two adjacent encoding modules for further encoding; Based on the second feature vector, the recognition result of the target image is determined; The first feature vector is encoded sequentially through each of the encoding modules in the self-attention network model according to the sequence of encoding modules and the first fusion rule to obtain the second feature vector, specifically including: According to the order of the encoding modules and the method of equally dividing the number of all encoding modules, the encoding modules are divided into at least two encoding sets, and each encoding set includes at least one of the encoding modules; Determine at least one pair of adjacent target encoding sets from the encoding set; The target position for fusing the first feature vector is defined as the position between the last encoding module of the first target encoding set and the first encoding module of the second target encoding set in the two adjacent target encoding sets. At each target location, the feature vectors encoded by the previous target encoding set in two adjacent target encoding sets are fused according to the second fusion rule of the target location to obtain the second feature vector. The second fusion rule for each target location is a specific rule for fusing feature vectors preset according to the actual scenario requirements, which means that the feature vectors are fused at the target location according to the preset rule.
2. The method according to claim 1, wherein the target location includes at least two, and in two adjacent target locations, the width or height of the first feature vector of the second target location is half the width or height of the first feature vector of the first target location.
3. The method according to any one of claims 1-2, wherein the target positions include at least two, and in two adjacent target positions, the first target position is fused according to width and the second target position is fused according to height, or the first target position is fused according to height and the second target position is fused according to width.
4. The method according to any one of claims 1-2, wherein the second feature vector comprises at least two, and the step of determining the recognition result of the target image based on the second feature vector comprises: The second feature vector is fused, and the number of the second feature vector after fusion is 1. Based on the fused second feature vector, a feature map corresponding to the target image is generated; The feature map is identified to obtain the recognition result of the target image.
5. The method according to any one of claims 1-2, wherein generating a feature map corresponding to the target image based on the fused second feature vector comprises: The second feature vector is subjected to global average pooling using a self-attention network model to obtain the feature map corresponding to the target image.
6. The method according to any one of claims 1-2, wherein the self-attention network model includes any one of the following: an image classification model, an image detection model, and an image localization model.
7. An image recognition device, the device comprising: A vectorization unit is used to vectorize the target image to obtain at least two first feature vectors; The encoding unit is used to encode the first feature vector sequentially through each of the encoding modules in the self-attention network model according to the sequence of encoding modules and the first fusion rule, to obtain the second feature vector; The encoding module includes at least two modules, and the first fusion rule includes: at the target position between at least one set of two adjacent encoding modules, fusing the feature vectors encoded by the first encoding module in the two adjacent encoding modules, wherein the number of fused feature vectors is less than the number of feature vectors encoded by the first encoding module, and inputting the fused feature vectors into the second encoding module in the two adjacent encoding modules for further encoding; A determining unit is configured to determine the recognition result of the target image based on the second feature vector; A partitioning unit is used to divide the encoding modules into at least two encoding sets according to the order of the encoding modules and the average number of all encoding modules, wherein each encoding set includes at least one of the encoding modules; The determining unit is further configured to determine at least one pair of adjacent target coding sets from the coding set; The determining unit is further configured to use the encoding module at the end of the first target encoding set and the encoding module at the beginning of the second target encoding set as the target position for fusing the first feature vector in the two adjacent target encoding sets. The encoding unit is specifically used for: At each target location, the feature vectors encoded by the previous target encoding set in two adjacent target encoding sets are fused according to the second fusion rule of the target location to obtain the second feature vector. The second fusion rule for each target location is a specific rule for fusing feature vectors preset according to the actual scenario requirements, which means that the feature vectors are fused at the target location according to the preset rule.
8. The apparatus according to claim 7, wherein the target positions include at least two, and in two adjacent target positions, the width or height of the first feature vector of the second target position is half the width or height of the first feature vector of the first target position.
9. The apparatus according to any one of claims 7-8, wherein the target positions include at least two, and in two adjacent target positions, the first target position is merged according to width and the second target position is merged according to height, or the first target position is merged according to height and the second target position is merged according to width.
10. The apparatus according to any one of claims 7-8, wherein the second feature vector comprises at least two, and the determining unit is specifically used for: The second feature vector is fused, and the number of the second feature vector after fusion is 1. Based on the fused second feature vector, a feature map corresponding to the target image is generated; The feature map is identified to obtain the recognition result of the target image.
11. The apparatus according to any one of claims 7-8, wherein the determining unit is specifically configured to: The second feature vector is subjected to global average pooling using the self-attention network model to obtain the feature map corresponding to the target image.
12. The apparatus according to any one of claims 7-8, wherein the self-attention network model comprises any one of the following: an image classification model, an image detection model, and an image localization model.
13. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-6.
14. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method according to any one of claims 1-6.
15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.