Image processing method and related apparatus thereof

Abstract

This application discloses an image processing method and related apparatus, which can effectively reduce the computational load of image processing, thereby shortening the overall processing time and improving the efficiency of image processing. The method of this application includes: after receiving N image blocks of a target image, the target model first evaluates the N image blocks to obtain evaluation values ​​for the N image blocks. Then, the target model uses the evaluation values ​​of the N image blocks as selection criteria to select M image blocks from the N image blocks. Next, the target model fuses the M image blocks to obtain a fusion result of the M image blocks. Finally, the target model performs a series of processing steps on the fusion result of the M image blocks to obtain the processed result of the target image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to an image processing method and related equipment. Background Technology

[0002] With the rapid development of computer technology, more and more fields are using neural network models from AI technology to complete various visual tasks. In order to explore neural network models with simpler structures, the multilayer perceptron (MLP) model has emerged. As a new type of visual backbone neural network, the MLP has achieved good results in a variety of visual tasks.

[0003] Currently, when processing an image in a vision task, the image is first divided into multiple image tokens, and these tokens are then input into a multilayer perceptron (MLP) model. The MLP model then fuses each image token to obtain a fused result. Subsequently, the MLP model performs a series of processing steps on the fused result to obtain the processed image, which can then be used to complete the vision task.

[0004] In the above process, since the visual multilayer perceptron model needs to fuse each image block, the amount of computation required is very large, which not only leads to the overall image processing time being too long, but also to the low efficiency of image processing. Summary of the Invention

[0005] This application provides an image processing method and related equipment, which can effectively reduce the computational load of image processing, thereby shortening the overall processing time and improving the efficiency of image processing.

[0006] A first aspect of this application provides an image processing method, which can be implemented through a target model, and the method includes:

[0007] When processing target images in a vision task, the target image can be divided into N image blocks, where N is a positive integer greater than 2.

[0008] After receiving N image patches from the target image, the target model can first evaluate each of the N image patches, thereby obtaining evaluation values ​​for the N image patches. It should be noted that the evaluation values ​​of the N image patches are used to indicate the importance of the content presented by the N image patches. For any one of the N image patches, the larger the evaluation value of the image patch, the more important the content presented by the image patch is; the smaller the evaluation value of the image patch, the less important the content presented by the image patch is.

[0009] After obtaining the evaluation values ​​of N image patches, the target model can select M image patches from the N image patches based on the magnitude relationship of the evaluation values, where M is a positive integer less than N and greater than or equal to 2. Therefore, among the N image patches constituting the target image, the M image patches selected by the target model can be considered as the more important parts of the target image.

[0010] After obtaining M image patches, the target model can perform a series of fusion operations on only the M image patches to obtain the fused result of the M image patches. After obtaining the fused result of the M image patches, the target model can perform a series of processing on the fused result of the M image patches to obtain the processed result of the target image. Then, the processed result of the target image can be used to complete the visual task.

[0011] As can be seen from the above method, after receiving N image blocks of the target image, the target model can first evaluate the N image blocks to obtain evaluation values ​​for each of them. Next, the target model can use these evaluation values ​​as selection criteria to select M image blocks from the N blocks. Then, the target model can fuse these M image blocks to obtain a fused result. Finally, the target model can perform a series of processing steps on the fused result to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image blocks indicate the importance of the content presented by each block, the M image blocks selected by the target model based on these evaluation values ​​are usually the more important parts of the target image. Therefore, in the process of obtaining the processed result of the target image, the target model only performs fusion operations on these M image blocks, without performing fusion operations on the remaining NM image blocks. This effectively reduces the computational load, thereby shortening the overall image processing time and improving the efficiency of image processing.

[0012] In one possible implementation, evaluating N image patches to obtain evaluation values ​​for the N image patches includes: performing a first fully connected layer on the N image patches to obtain first features of the N image patches; pooling the first features of the N image patches to obtain second features of the N image patches; and multiplying the first features and second features of the N image patches to obtain third features of the N image patches. The third features of the N image patches are then used as the evaluation values ​​for the N image patches. In the aforementioned implementation, after receiving N image patches, the target model can perform a first fully connected layer on the N image patches to obtain the first features of the N image patches. After obtaining the first features of the N image patches, the target model can pool the first features of the N image patches to obtain the second features of the N image patches. After obtaining the second features of the N image patches, the target model can multiply the first features and second features of the N image patches to obtain the third features of the N image patches. The third features of the N image patches can then be used as the evaluation values ​​for the N image patches.

[0013] In one possible implementation, N image patches form an X-row, Y-column image patch array. Based on evaluation values, determining M image patches from the N image patches includes: selecting the P image patches with the highest evaluation values ​​in the i-th row, i = 1, ..., X, M = XP, P ≥ 1; or selecting the K image patches with the highest evaluation values ​​in the j-th column, j = 1, ..., Y, M = YK, K ≥ 1. In the aforementioned implementation, the target model can select image patches as follows: After obtaining the evaluation values ​​of the N image patches, since the N image patches are presented in X-row format, the target model can select the P image patches with the highest evaluation values ​​in the first row, the second row, ..., and the X-th row. In this way, the target model can select a total of M = XP image patches horizontally. Of course, the target model can also select image patches in the following way: After obtaining the evaluation values ​​of N image patches, since the N image patches are presented in the form of Y columns of image patches, the target model can select the K image patches with the largest evaluation values ​​in the first column of image patches, the K image patches with the largest evaluation values ​​in the second column of image patches, ..., and the K image patches with the largest evaluation values ​​in the Y column of image patches. In this way, the target model can select a total of M = YK image patches vertically upwards.

[0014] In one possible implementation, the method further includes: weighted summation of the evaluation values ​​of N image patches with the first features of the N image patches to obtain fourth features of the N image patches; multiplying the fourth features of the N image patches with the evaluation values ​​of M image patches to obtain fifth features of the M image patches; fusing the M image patches to obtain a fusion result includes: concatenating the M image patches with the fifth features of the M image patches to obtain sixth features of the M image patches; performing a fully connected operation on the sixth features of the M image patches to obtain seventh features of the M image patches, with the seventh features of the M image patches serving as the fusion result of the M image patches. In the aforementioned implementation, after obtaining the evaluation values ​​of the N image patches, the target model can also use the evaluation values ​​of the N image patches as weights, and use these weights to weighted summation of the first features of the N image patches to obtain fourth features of the N image patches. After obtaining the fourth features of N image patches, the target model can multiply the fourth features of the N image patches with the evaluation values ​​of M image patches to obtain the fifth features of the M image patches. After obtaining the fifth features of the N image patches, the target model can concatenate the fifth features of the M image patches to obtain the sixth features of the M image patches. After obtaining the fifth features of the N image patches, the target model performs a fully connected operation on the sixth features of the M image patches to obtain the seventh features of the M image patches. The seventh features of the M image patches are then used as the fusion result of the M image patches.

[0015] In one possible implementation, obtaining the target image processing result based on the fusion result of M image patches includes: performing a second fully connected layer on N image patches to obtain the eighth features of the N image patches; performing a weighted sum of the fusion result of the M image patches and the eighth features of the M image patches to obtain the ninth features of the M image patches; performing a weighted sum of the eighth features of the NM image patches (excluding the M image patches) and the NM image patches to obtain the ninth features of the NM image patches; and processing the ninth features of the N image patches to obtain the processing result of the target image. In the aforementioned implementation, the target model can also perform a second fully connected layer on the N image patches to obtain the eighth features of the N image patches. After obtaining the eighth features of the N image patches, the target model can also use preset weights to perform a weighted sum of the fusion result of the M image patches and the eighth features of the M image patches to obtain the ninth features of the M image patches. After obtaining the eighth features of N image patches, the target model can use preset weights to perform a weighted summation of the eighth features of the remaining NM image patches (excluding M patches) to obtain the ninth features of the NM image patches. After obtaining the ninth features of the N image patches, the target model further processes these ninth features to obtain the processed result of the target image.

[0016] In one possible implementation, the aforementioned processing includes at least one of the following: normalization, aggregation, or addition. In this implementation, the target model can superimpose the ninth features of N image patches with the N image patches to obtain the tenth features of the N image patches. Next, the target model can normalize the tenth features of the N image patches to obtain the eleventh features of the N image patches. Then, the target model can aggregate the eleventh features of the N image patches along the channel dimension to obtain the twelfth features of the N image patches. Finally, the target model can superimpose the twelfth features of the N image patches with the ninth features of the N image patches to obtain the processed result of the target image.

[0017] In one possible implementation, before evaluating N image patches and obtaining their evaluation values, the method further includes normalizing the N image patches to obtain normalized N image patches. In the aforementioned implementation, the target model may also first normalize the N image patches to obtain normalized N image patches, and then perform various processing on the normalized N image patches to obtain the ninth feature of the normalized N image patches.

[0018] A second aspect of this application provides a model training method, comprising: inputting a target image into a model to be trained to obtain a processing result of the target image, wherein the model to be trained is used to: acquire N image patches of the target image; evaluate the N image patches to obtain evaluation values ​​of the N image patches, wherein the evaluation values ​​of the N image patches are used to indicate the importance of the content presented by the N image patches; determine M image patches from the N image patches based on the evaluation values ​​of the N image patches, where N > M ≥ 2; fuse the M image patches to obtain a fusion result of the M image patches; obtain a processing result of the target image based on the fusion result of the M image patches; obtain a target loss based on the processing result and the actual processing result of the target image; and update the parameters of the model to be trained based on the target loss until the model training conditions are met to obtain a target model.

[0019] The target model trained using the above method possesses image processing capabilities. Specifically, after receiving N image patches from the target image, the target model first evaluates each of the N image patches to obtain evaluation values. Next, the target model uses these evaluation values ​​as selection criteria to select M image patches from the N patches. Then, the target model fuses these M image patches to obtain a fused result. Finally, the target model performs a series of processing steps on the fused result to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image patches indicate the importance of the content presented by each patch, the M image patches selected by the target model based on these evaluation values ​​are usually the more important parts of the target image. Therefore, in obtaining the processed result of the target image, the target model only performs fusion operations on these M image patches, without fusing the remaining NM image patches, which effectively reduces the computational load, thereby shortening the overall image processing time and improving image processing efficiency.

[0020] In one possible implementation, the model to be trained is used to: perform a first fully connected operation on N image patches to obtain first features of the N image patches; pool the first features of the N image patches to obtain second features of the N image patches; multiply the first features of the N image patches and the second features of the N image patches to obtain third features of the N image patches, and use the third features of the N image patches as evaluation values ​​of the N image patches.

[0021] In one possible implementation, N image patches form an X-row, Y-column image patch array. The model to be trained is used to: select the P image patches with the largest evaluation values ​​in the i-th row of image patches, i = 1, ..., X, M = XP, P ≥ 1; or, select the K image patches with the largest evaluation values ​​in the j-th column of image patches, j = 1, ..., Y, M = YK, K ≥ 1.

[0022] In one possible implementation, the model to be trained is further configured to: perform a weighted summation of the evaluation values ​​of N image patches with the first features of the N image patches to obtain the fourth features of the N image patches; multiply the fourth features of the N image patches with the evaluation values ​​of M image patches to obtain the fifth features of the M image patches; the model to be trained is configured to: concatenate the M image patches with the fifth features of the M image patches to obtain the sixth features of the M image patches; perform a fully connected operation on the sixth features of the M image patches to obtain the seventh features of the M image patches, and the seventh features of the M image patches are used as the fusion result of the M image patches.

[0023] In one possible implementation, the model to be trained is used to: perform a second fully connected layer on N image patches to obtain the eighth features of the N image patches; perform a weighted summation of the fusion result of M image patches and the eighth features of the M image patches to obtain the ninth features of the M image patches; perform a weighted summation of the eighth features of the NM image patches other than the M image patches to obtain the ninth features of the NM image patches; and process the ninth features of the N image patches to obtain the processing result of the target image.

[0024] In one possible implementation, the processing includes at least one of the following: normalization, aggregation, or addition.

[0025] In one possible implementation, the model to be trained is also used to: normalize N image patches to obtain normalized N image patches.

[0026] A third aspect of this application provides an image processing apparatus comprising a target model. The apparatus includes: a first acquisition module for acquiring N image blocks of a target image; an evaluation module for evaluating the N image blocks to obtain evaluation values ​​for the N image blocks, the evaluation values ​​of the N image blocks indicating the importance of the content presented by the N image blocks; a determination module for determining M image blocks from the N image blocks based on the evaluation values ​​of the N image blocks, where N > M ≥ 2; a fusion module for fusing the M image blocks to obtain a fusion result of the M image blocks; and a second acquisition module for acquiring a processing result of the target image based on the fusion result of the M image blocks.

[0027] As can be seen from the above apparatus, after receiving N image blocks of the target image, the target model can first evaluate the N image blocks to obtain evaluation values ​​for the N image blocks. Next, the target model can use the evaluation values ​​of the N image blocks as selection criteria to select M image blocks from the N image blocks. Then, the target model can fuse the M image blocks to obtain a fused result of M image blocks. Finally, the target model can perform a series of processing steps on the fused result of the M image blocks to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image blocks are used to indicate the importance of the content presented by the N image blocks, the M image blocks selected by the target model based on the evaluation values ​​are usually the more important parts of the target image. Therefore, in the process of obtaining the processed result of the target image, the target model only performs fusion operations on these M image blocks, and does not perform fusion operations on the remaining NM image blocks, which can effectively reduce the amount of computation, thereby shortening the overall image processing time and improving the efficiency of image processing.

[0028] In one possible implementation, the evaluation module is configured to: perform a first fully connected operation on N image patches to obtain first features of the N image patches; pool the first features of the N image patches to obtain second features of the N image patches; multiply the first features and second features of the N image patches to obtain third features of the N image patches, and use the third features of the N image patches as evaluation values ​​of the N image patches.

[0029] In one possible implementation, N image blocks form an X-row, Y-column image block array. The determining module is used to: select the P image blocks with the largest evaluation values ​​in the i-th row of image blocks, i = 1, ..., X, M = XP, P ≥ 1; or select the K image blocks with the largest evaluation values ​​in the j-th column of image blocks, j = 1, ..., Y, M = YK, K ≥ 1.

[0030] In one possible implementation, the device further includes: a summation module for weighted summation of the evaluation values ​​of N image blocks with the first features of the N image blocks to obtain a fourth feature of the N image blocks; a multiplication module for multiplying the fourth features of the N image blocks with the evaluation values ​​of M image blocks to obtain a fifth feature of the M image blocks; and a fusion module for: concatenating the M image blocks with the fifth features of the M image blocks to obtain a sixth feature of the M image blocks; performing a full connection on the sixth features of the M image blocks to obtain a seventh feature of the M image blocks, wherein the seventh feature of the M image blocks is used as the fusion result of the M image blocks.

[0031] In one possible implementation, the second acquisition module is configured to: perform a second fully connected operation on N image blocks to obtain the eighth features of the N image blocks; perform a weighted summation of the fusion result of M image blocks and the eighth features of the M image blocks to obtain the ninth features of the M image blocks; perform a weighted summation of the eighth features of the NM image blocks other than the M image blocks to obtain the ninth features of the NM image blocks; and process the ninth features of the N image blocks to obtain the processing result of the target image.

[0032] In one possible implementation, the processing includes at least one of the following: normalization, aggregation, or addition.

[0033] In one possible implementation, the device further includes a normalization module for normalizing N image blocks to obtain normalized N image blocks.

[0034] A fourth aspect of this application provides a model training apparatus, comprising: an input module for inputting a target image into a model to be trained to obtain a processing result of the target image, wherein the model to be trained is configured to: acquire N image patches of the target image; evaluate the N image patches to obtain evaluation values ​​for the N image patches, the evaluation values ​​of the N image patches indicating the importance of the content presented by the N image patches; determine M image patches from the N image patches based on the evaluation values ​​of the N image patches, where N > M ≥ 2; fuse the M image patches to obtain a fusion result of the M image patches; and acquire a processing result of the target image based on the fusion result of the M image patches; an acquisition module for acquiring a target loss based on the processing result and the actual processing result of the target image; and an update module for updating the parameters of the model to be trained based on the target loss until the model training conditions are met to obtain a target model.

[0035] The target model trained by the aforementioned device possesses image processing capabilities. Specifically, after receiving N image blocks of a target image, the target model first evaluates the N image blocks to obtain evaluation values ​​for each block. Next, the target model uses these evaluation values ​​as selection criteria to select M image blocks from the N blocks. Then, the target model fuses the M image blocks to obtain a fused result. Finally, the target model performs a series of processing steps on the fused result to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image blocks indicate the importance of the content presented by each block, the M image blocks selected by the target model based on these evaluation values ​​are typically the more important parts of the target image. Therefore, in obtaining the processed result of the target image, the target model only performs fusion operations on these M image blocks, without fusing the remaining NM image blocks. This effectively reduces the computational load, thereby shortening the overall image processing time and improving image processing efficiency.

[0036] In one possible implementation, the model to be trained is used to: perform a first fully connected operation on N image patches to obtain first features of the N image patches; pool the first features of the N image patches to obtain second features of the N image patches; multiply the first features of the N image patches and the second features of the N image patches to obtain third features of the N image patches, and use the third features of the N image patches as evaluation values ​​of the N image patches.

[0037] In one possible implementation, N image patches form an X-row, Y-column image patch array. The model to be trained is used to: select the P image patches with the largest evaluation values ​​in the i-th row of image patches, i = 1, ..., X, M = XP, P ≥ 1; or, select the K image patches with the largest evaluation values ​​in the j-th column of image patches, j = 1, ..., Y, M = YK, K ≥ 1.

[0038] In one possible implementation, the model to be trained is further configured to: perform a weighted summation of the evaluation values ​​of N image patches with the first features of the N image patches to obtain the fourth features of the N image patches; multiply the fourth features of the N image patches with the evaluation values ​​of M image patches to obtain the fifth features of the M image patches; the model to be trained is configured to: concatenate the M image patches with the fifth features of the M image patches to obtain the sixth features of the M image patches; perform a fully connected operation on the sixth features of the M image patches to obtain the seventh features of the M image patches, and the seventh features of the M image patches are used as the fusion result of the M image patches.

[0039] In one possible implementation, the model to be trained is used to: perform a second fully connected layer on N image patches to obtain the eighth features of the N image patches; perform a weighted summation of the fusion result of M image patches and the eighth features of the M image patches to obtain the ninth features of the M image patches; perform a weighted summation of the eighth features of the NM image patches other than the M image patches to obtain the ninth features of the NM image patches; and process the ninth features of the N image patches to obtain the processing result of the target image.

[0040] In one possible implementation, the processing includes at least one of the following: normalization, aggregation, or addition.

[0041] In one possible implementation, the model to be trained is also used to: normalize N image patches to obtain normalized N image patches.

[0042] A fifth aspect of this application provides an image processing apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the image processing apparatus performs the method described in the first aspect or any possible implementation thereof.

[0043] A sixth aspect of this application provides a model training apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training apparatus performs the method described in the second aspect or any possible implementation thereof.

[0044] A seventh aspect of this application provides a circuit system including a processing circuit configured to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0045] An eighth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory to cause the processor to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0046] In one possible implementation, the processor is coupled to the memory via an interface.

[0047] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0048] A ninth aspect of this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0049] A tenth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0050] In this embodiment, after receiving N image blocks of the target image, the target model first evaluates the N image blocks to obtain evaluation values ​​for each block. Then, the target model uses these evaluation values ​​as selection criteria to select M image blocks from the N blocks. Next, the target model fuses the M image blocks to obtain a fused result. Finally, the target model performs a series of processing steps on the fused result to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image blocks indicate the importance of the content presented by each block, the M image blocks selected by the target model based on these evaluation values ​​are usually the more important parts of the target image. Therefore, in obtaining the processed result of the target image, the target model only performs fusion operations on these M image blocks, without fusing the remaining NM image blocks. This effectively reduces the computational load, shortens the overall image processing time, and improves the efficiency of image processing. Attached Figure Description

[0051] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0052] Figure 2aA schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0053] Figure 2b This is another schematic diagram of the image processing system provided in the embodiments of this application;

[0054] Figure 2c A schematic diagram of an image processing device provided in an embodiment of this application;

[0055] Figure 3 A schematic diagram of the system 100 architecture provided in the embodiments of this application;

[0056] Figure 4 A schematic diagram of the structure of the target model provided in the embodiments of this application;

[0057] Figure 5 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0058] Figure 6 A schematic diagram of the structure of a dynamic image block unit provided in an embodiment of this application;

[0059] Figure 7 This is another structural schematic diagram of the dynamic image block unit provided in the embodiments of this application;

[0060] Figure 8 Another structural schematic diagram of the target model provided in the embodiments of this application;

[0061] Figure 9 Another structural schematic diagram of the target model provided in the embodiments of this application;

[0062] Figure 10 A schematic flowchart of the model training method provided in the embodiments of this application;

[0063] Figure 11 A schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0064] Figure 12 A schematic diagram of the structure of the model training apparatus provided in the embodiments of this application;

[0065] Figure 13 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0066] Figure 14 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0067] Figure 15 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0068] This application provides an image processing method and related equipment, which can effectively reduce the computational load of image processing, thereby shortening the overall processing time and improving the efficiency of image processing.

[0069] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0070] With the rapid development of computer technology, more and more fields such as autonomous driving and identity recognition are using neural network models from AI technology to complete various visual tasks, such as image classification, object detection, and instance segmentation. To explore more concise neural network models, the visual multilayer perceptron (MLP) has emerged. As a novel type of visual backbone neural network, the MLP has achieved good results in various visual tasks.

[0071] Currently, when processing an image in a visual task, the image is first divided into multiple image patches, which are then input into a multilayer perceptron (MLP) model. After obtaining these image patches, the MLP model fuses any one patch with its surrounding patches to obtain a fused image. This process is repeated for the remaining image patches, allowing the MLP model to obtain fused results from multiple image patches. Subsequently, the MLP model performs a series of processing steps on the fused results to obtain the processed image, which can then be used to complete the visual task.

[0072] In the above process, since the visual multilayer perceptron model needs to fuse each image block, the amount of computation required is very large, which not only leads to the overall image processing time being too long, but also to the low efficiency of image processing.

[0073] Furthermore, in related technologies, other neural network models can be used to filter out a portion of the image patches from multiple image blocks in the image, and then the remaining image patches can be input into a visual multilayer perceptron model for processing to obtain the processed image result. Although this can appropriately reduce the computational load of the visual multilayer perceptron model, it requires the introduction of some additional neural network models, and additional work is required in both the model training and model application phases.

[0074] To address the aforementioned problems, this application provides an image processing method that can be implemented in conjunction with artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. Using artificial intelligence for data processing is a common application of AI.

[0075] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 This is a structural diagram illustrating the main framework of artificial intelligence. The following explanation of the AI ​​framework is based on two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0076] (1) Infrastructure

[0077] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0078] (2) Data

[0079] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0080] (3) Data processing

[0081] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0082] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0083] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0084] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0085] (4) General ability

[0086] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0087] (5) Smart Products and Industry Applications

[0088] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0089] The following sections will introduce several application scenarios for this application.

[0090] Figure 2a This is a schematic diagram of an image processing system provided in an embodiment of this application. The image processing system includes a user device and a data processing device. The user device includes smart terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiator of image processing; as the initiator of image processing requests, requests are typically initiated by the user through the user device.

[0091] The aforementioned data processing equipment can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing equipment receives image processing requests from smart terminals through an interactive interface, and then performs image processing methods such as machine learning, deep learning, search, reasoning, and decision-making through a storage device and a data processing processor. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0092] exist Figure 2a In the image processing system shown, the user equipment can receive user instructions. For example, the user equipment can acquire an image for a visual task input / selected by the user, and then send a request to the data processing device. This causes the data processing device to perform image processing on the image acquired by the user equipment, thereby obtaining the corresponding processing result for that image. For instance, the user equipment can acquire an image input by the user and then send an image processing request to the data processing device. This causes the data processing device to perform a series of processes on the image, thereby obtaining the processing result, such as the classification result of the image, the detection box region surrounding the target object in the image, etc. Therefore, the processing result of the image can be used to complete the user's visual task.

[0093] exist Figure 2a In this context, the data processing device can execute the image processing method of the embodiments of this application.

[0094] Figure 2b This is another schematic diagram of the image processing system provided in the embodiments of this application. Figure 2b In this context, the user equipment (UE) directly functions as a data processing device. This UE can directly acquire input from the user and process it directly through its own hardware. The specific process is similar to... Figure 2a Similar to the description above, it will not be repeated here.

[0095] exist Figure 2b In the image processing system shown, the user equipment can receive user instructions. For example, the user equipment can acquire an image input by the user and then perform a series of processing on the image to obtain the processing result of the image, such as the classification result of the image, the detection box region surrounding the target object in the image, etc. Therefore, the processing result of the image can be used to complete the user's visual task.

[0096] exist Figure 2b In this application, the user equipment itself can execute the image processing method of the embodiments of this application.

[0097] Figure 2cThis is a schematic diagram of an image processing device provided in an embodiment of this application.

[0098] The above Figure 2a and Figure 2b The user equipment in the context can specifically be Figure 2c Local device 301 or local device 302 in the system. Figure 2a The data processing equipment in the middle can specifically be Figure 2c The execution device 210 in the process includes a data storage system 250 that can store the data to be processed by the execution device 210. The data storage system 250 can be integrated into the execution device 210 or set up in the cloud or on other network servers.

[0099] Figure 2a and Figure 2b The processor in the image can be trained on data using neural network models or other models (e.g., support vector machine-based models) for machine learning / deep learning, and then use the trained or learned models to perform image processing applications on the image to obtain the corresponding processing results.

[0100] Figure 3 A schematic diagram of the system 100 architecture provided in this application embodiment, in Figure 3 In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data in this embodiment may include various scheduled tasks, callable resources, and other parameters.

[0101] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0102] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0103] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These target models / rules can then be used to achieve the aforementioned objectives or complete the aforementioned tasks, thereby providing the user with the required results. The training data can be stored in the database 130 and originates from training samples collected by the data acquisition device 160.

[0104] exist Figure 3 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0105] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this context, the data storage system 150 is an external memory relative to the execution device 110. However, in other cases, the data storage system 150 can also be placed within the execution device 110. For example... Figure 3 As shown, a neural network can be trained using training device 120.

[0106] This application also provides a chip including a neural network processor (NPU). This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rules.

[0107] The Neural Processing Unit (NPU) is a coprocessor mounted on the main central processing unit (CPU) (host CPU), where tasks are assigned by the CPU. The core of the NPU is the computation circuitry, which is controlled by a controller to retrieve data from memory (weight memory or input memory) and perform calculations.

[0108] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0109] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory and caches it in each PE (Process Equipment) of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator.

[0110] Vector computation units can further process the output of computational circuits, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, vector computation units can be used for computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0111] In some implementations, the vector computation unit can store the processed output vector into a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, for example, for use in subsequent layers of a neural network.

[0112] The unified memory is used to store input data and output data.

[0113] The weight data is directly transferred from the external memory to the input memory and / or unified memory, stored in the weight memory, and stored in the unified memory to the external memory through the direct memory access controller (DMAC).

[0114] The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC, and instruction fetch memory via a bus.

[0115] The instruction fetch buffer, connected to the controller, is used to store the instructions used by the controller.

[0116] The controller is used to invoke instructions cached in the memory to control the operation of the computing accelerator.

[0117] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, while external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDRSDRAM), high bandwidth memory (HBM), or other readable and writable memories.

[0118] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0119] (1) Neural Network

[0120] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0121]

[0122] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0123] The work of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 by +b, and operation 5 by a(). The term "space" is used here because the objects being classified are not individual things, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight vector, and each value in this vector represents the weight value of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above; that is, the weights W of each layer control how the space is transformed. The purpose of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control the transformation space, and more specifically, learning the weight matrix.

[0124] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0125] (2) Backpropagation algorithm

[0126] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0127] The method provided in this application is described below from the perspectives of neural network training and neural network application.

[0128] The model training method provided in this application involves the processing of data sequences and can be applied to data training, machine learning, deep learning, and other methods. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (e.g., the target image in the model training method provided in this application), ultimately obtaining a trained neural network (e.g., the target model in the model training method provided in this application). Furthermore, the image processing method provided in this application can utilize the trained neural network to input input data (e.g., the target image in the image processing method provided in this application) into the trained neural network to obtain output data (e.g., the processing result of the target image in the image processing method provided in this application). It should be noted that the model training method and image processing method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of a whole process: such as the model training stage and the model application stage.

[0129] The image processing method provided in this application embodiment can be implemented by a novel visual multilayer perceptron model, which will be referred to as the target model. The target model can have multiple structures, and the first type of target model will be introduced below. Figure 4 A schematic diagram of the structure of the target model provided in the embodiments of this application, such as Figure 4 As shown, the target model contains a dynamic multilayer perceptron (MLP) module, which includes a dynamic-token unit and a processing unit. To understand... Figure 4 The workflow of the target model shown below, in conjunction with... Figure 5 This workflow will be described. Figure 5 This is a schematic flowchart of an image processing method provided in an embodiment of this application, as shown below. Figure 5 As shown, the method includes:

[0130] 501. Obtain N image blocks of the target image.

[0131] In this embodiment, when it is necessary to process the target image in the visual task, the target image can be divided into N image blocks. The N image blocks can be presented by an X-row Y-column image block array (that is, the image block array contains X rows of image blocks, or the image block array contains Y columns of image blocks). N is a positive integer greater than 2, and X and Y are positive integers greater than or equal to 2.

[0132] After obtaining N image patches of the target image, the N image patches can be input into the target model (a trained neural network model) to perform a series of processing on the N image patches.

[0133] 502. Evaluate N image patches to obtain evaluation values ​​for N image patches. The evaluation values ​​of N image patches are used to indicate the importance of the content presented by the N image patches.

[0134] After receiving N image patches from the target image, the target model can first evaluate each of the N image patches, thereby obtaining evaluation values ​​for the N image patches. It should be noted that the evaluation values ​​of the N image patches are used to indicate the importance of the content presented by the N image patches (which can also be understood as the richness of the content presented by the N image patches, or the degree of contribution of the N image patches to the visual task, etc.). For any one of the N image patches, the larger the evaluation value of the image patch, the more important the content presented by the image patch is; the smaller the evaluation value of the image patch, the less important the content presented by the image patch is.

[0135] Specifically, the target model can obtain the evaluation values ​​of N image patches in the following way:

[0136] (1) After receiving N image blocks, the dynamic image block unit of the target model can perform a first full connection (FC) on the N image blocks to obtain the first features of the N image blocks. It should be noted that the height of the whole formed by the N image blocks (i.e., the target image) is the same as the height of the whole formed by the first features of the N image blocks (i.e., the first feature map), and the width of the whole formed by the N image blocks is the same as the width of the whole formed by the first features of the N image blocks. However, for any one of the N image blocks, the number of channels of the image block is more than the number of channels of the first feature of the image block.

[0137] For example, such as Figure 6 As shown ( Figure 6(This is a schematic diagram of a dynamic image block unit provided in an embodiment of this application). Assume the target image is divided into 36 image blocks arranged in a 6x6 column format. Therefore, the total height of these 36 image blocks is H = 6, the total width of these 36 image blocks is W = 6, and the number of channels for each of these 36 image blocks is C. After receiving these 36 image blocks, the horizontal predictor of the dynamic image block unit can perform a fully connected operation on these 36 image blocks using the following formula to obtain the horizontal features (the aforementioned first feature), i.e., the horizontal feature map (the aforementioned first feature map):

[0138] X h =FC(X,W) h (2)

[0139] In the above formula, X is the target image, and W... h The parameters used for full connection, X h This is a horizontal feature map, containing the horizontal features of these 36 image patches. It should be noted that X... h The height is H, X h The width of the horizontal feature is W, X h The number of channels is C / 2.

[0140] Meanwhile, the vertical predictor of the dynamic image patch unit can also perform a similar operation on these 36 image patches, thus obtaining vertical features. Figure X v (The aforementioned first feature map) contains the vertical features of 36 image patches (the aforementioned first feature). It should be noted that X... v The height is H, X v The width is W, X v The number of channels is C / 2.

[0141] (2) After obtaining the first features of N image blocks, the dynamic image block unit can perform pooling on the first features of the N image blocks to obtain the second features of the N image blocks. It should be noted that after pooling (dimensionality reduction), two situations may occur. In the first situation, the height of the whole formed by the first features of the N image blocks is the same as the height of the whole formed by the second features of the N image blocks (i.e., the second feature map), and the width of the whole formed by the first features of the N image blocks is larger than the width of the whole formed by the first features of the N image blocks. However, for any image block among the N image blocks, the number of channels of the first feature of that image block is the same as the number of channels of the second feature of that image block. In the second situation, the height of the whole formed by the first features of the N image blocks is larger than the height of the whole formed by the second features of the N image blocks, and the width of the whole formed by the first features of the N image blocks is the same as the width of the whole formed by the first features of the N image blocks. However, for any image block among the N image blocks, the number of channels of the first feature of that image block is the same as the number of channels of the second feature of that image block.

[0142] As in the example above, we get X. h Then, the horizontal predictor can predict X. h Pooling is performed to obtain the horizontal labeled feature map. (The aforementioned second feature map) contains the horizontal labeling features (the aforementioned second feature) of these 36 images. It should be noted that... The height is H. The width is 1. The number of channels is C / 2.

[0143] At the same time, X was obtained v Then, the vertical predictor can also predict X. v By performing similar operations, a vertically oriented marker feature map can be obtained. (The aforementioned second feature map) contains the vertically labeled features of 36 image patches (the aforementioned second feature). It should be noted that... The height is 1. The width is W. The number of channels is C / 2.

[0144] (3) After obtaining the second features of N image blocks, the dynamic image block unit can multiply the first features and the second features of the N image blocks (e.g., dot product, etc.) to obtain the third features of the N image blocks. The third features of the N image blocks can then be used as the evaluation value of the N image blocks. It should be noted that the height of the whole formed by the first features of the N image blocks is the same as the height of the whole formed by the third features of the N image blocks (i.e., the third feature map), and the width of the whole formed by the first features of the N image blocks is the same as the width of the whole formed by the third features of the N image blocks. However, for any image block among the N image blocks, the number of channels of the first feature of that image block is greater than the number of channels of the third feature of that image block.

[0145] As in the example above, we get Then, the horizontal predictor can be calculated using the following formula. and X h Perform a dot product to obtain the horizontal evaluation value map:

[0146]

[0147] In the above formula, S h This is the horizontal evaluation value map, containing the horizontal evaluation values ​​of these 36 image patches. It should be noted that S... h The height is H, S h The width is W, S h The number of channels is 1.

[0148] At the same time, the vertical predictor can also predict X. v as well as By performing a similar operation, the vertical evaluation value map S can be obtained. v This includes the vertical evaluation values ​​for these 36 image patches. It should be noted that S... v The height is H, S v The width is W, S v The number of channels is 1.

[0149] 503. Based on the evaluation values ​​of N image blocks, determine M image blocks from the N image blocks, where N > M ≥ 2.

[0150] After obtaining the evaluation values ​​of N image patches, the target model can select M image patches from the N image patches based on the magnitude relationship of the evaluation values, where M is a positive integer less than N and greater than or equal to 2. Therefore, among the N image patches constituting the target image, the M image patches selected by the target model can be considered as the more important parts of the target image, while the remaining NM unselected image patches can be considered as the less important parts of the target image.

[0151] Specifically, the target model can select M image patches in the following way:

[0152] (1) After obtaining the evaluation values ​​of N image blocks, since the N image blocks are presented in the form of X rows of image blocks (each row of image blocks includes Y image blocks), the dynamic image block unit can select the P image blocks with the largest evaluation values ​​(P is a positive integer greater than or equal to 1) in the first row of image blocks according to a certain image block selection ratio ρ = P / Y, select the P image blocks with the largest evaluation values ​​in the second row of image blocks, ..., and select the P image blocks with the largest evaluation values ​​in the Xth row of image blocks. In this way, the dynamic image block unit can select a total of M = XP image blocks horizontally.

[0153] (2) After obtaining the evaluation values ​​of N image blocks, since the N image blocks are presented in the form of Y columns of image blocks (each column of image blocks includes X image blocks), the dynamic image block unit can select the K image blocks with the largest evaluation values ​​(K is a positive integer greater than or equal to 1) in the first column of image blocks according to a certain image block selection ratio ρ = K / X, select the K image blocks with the largest evaluation values ​​in the second column of image blocks, ..., and select the K image blocks with the largest evaluation values ​​in the Y column of image blocks. In this way, the dynamic image block unit can select a total of M = YK image blocks vertically upwards.

[0154] As in the example above, we obtain S. h Since these 36 image patches are presented in 6 rows, the horizontal predictor can select the 3 image patches with the highest evaluation values ​​from each row according to ρ = 0.5. Therefore, a total of 18 image patches can be obtained horizontally. The set of numbers of these image patches can be denoted as:

[0155] id h =Top-ρ(S) h ,ρ) (4)

[0156] In the above formula, id h This is a set of numbered image patches selected horizontally.

[0157] The set of image patches selected horizontally can be denoted as:

[0158]

[0159] In the above formula, The set of image patches selected horizontally. For a given image patch selected horizontally, u is the patch number.

[0160] At the same time, S was obtained vSince these 36 image patches are presented in 6 columns, the horizontal predictor can select the 3 image patches with the highest evaluation values ​​from each column according to ρ = 0.5, thus obtaining a total of 18 image patches in the vertical direction. Similarly, the set of image patches selected in the vertical direction can be denoted as...

[0161] 504. Fuse M image patches to obtain the fused result of M image patches.

[0162] After obtaining M image patches, the target model can perform a series of fusion operations on the M image patches without performing these fusion operations on the NM image patches, thereby obtaining the fusion result of the M image patches.

[0163] Specifically, the target model can obtain the fusion result of M image patches in the following way:

[0164] (1) After obtaining the evaluation values ​​of N image blocks, the dynamic image block unit can also use the evaluation values ​​of the N image blocks as weights to perform a weighted summation of the first features of the N image blocks, thereby obtaining the fourth features of the N image blocks. It should be noted that after the weighted summation (dimensionality reduction), two situations may occur. In the first case, the height of the whole formed by the first features of the N image blocks is the same as the height of the whole formed by the fourth features of the N image blocks (i.e., the fourth feature map), and the width of the whole formed by the first features of the N image blocks is larger than the width of the whole formed by the fourth features of the N image blocks. However, for any image block among the N image blocks, the number of channels of the first feature of that image block is the same as the number of channels of the fourth feature of that image block. In the second case, the height of the whole formed by the first features of the N image blocks is greater than the height of the whole formed by the fourth features of the N image blocks, and the width of the whole formed by the first features of the N image blocks is the same as the width of the whole formed by the fourth features of the N image blocks. However, for any one of the N image blocks, the number of channels of the first feature of that image block is the same as the number of channels of the fourth feature of that image block.

[0165] As in the example above, we obtain S. h Then, the horizontal predictor can be used to predict S using the following formula. h and X h We perform a weighted summation to obtain the global feature map in the horizontal direction:

[0166]

[0167] In the above formula, The horizontal global feature map (the aforementioned fourth feature map) contains the horizontal global features (the aforementioned fourth feature) of these 36 image patches. Let be the evaluation value of the image patch in the i-th row. Let H = 6, representing the horizontal feature of the i-th row of the image patch. It should be noted that... The height is H. The width is 1. The number of channels is C / 2.

[0168] At the same time, the vertical predictor can also predict S v and X v Perform similar operations to obtain a vertical global feature map. (The aforementioned fourth feature map) contains the global features (the aforementioned fourth feature) of these 36 image patches in the vertical direction. It should be noted that... The height is 1. The width is W. The number of channels is C / 2.

[0169] (2) After obtaining the fourth features of N image blocks, the dynamic image block unit can multiply the fourth features of the N image blocks with the evaluation values ​​of M image blocks (e.g., dot product, etc.) to obtain the fifth features of M image blocks. It should be noted that the height of the whole formed by the M image blocks is the same as the height of the whole formed by the fifth features of the M image blocks (i.e., the fifth feature map), and the width of the whole formed by the M image blocks is the same as the width of the whole formed by the fifth features of the M image blocks. However, for any one of the M image blocks, the number of channels of that image block is greater than the number of channels of the fifth feature of that image block.

[0170] As in the example above, we get Then, the horizontal predictor can Sending to the horizontal mixer, similarly, the vertical predictor can also... Send to the horizontal mixer. Then, the horizontal mixer can be configured using the following formula... This process yields the first globally expanded feature map in the horizontal direction:

[0171]

[0172]

[0173] In the above formula, The first global extended feature map in the horizontal direction (the aforementioned fifth feature map) contains the first global extended feature (the aforementioned fifth feature) of the 18 image patches selected in the horizontal direction. This is the set of evaluation values ​​for selecting 18 image patches that appear horizontally. This is the evaluation value for selecting a specific image patch that appears horizontally. It should be noted that... The height is H. The width is W×ρ (i.e., 3). The number of channels is C / 2.

[0174] Next, the horizontal mixer can also be adjusted using the following formula. This process yields a second globally expanded feature map in the horizontal direction:

[0175]

[0176] In the above formula, The second global extended feature map in the horizontal direction (the aforementioned fifth feature map) contains the second global extended features (the aforementioned fifth feature) of 18 image patches selected horizontally. It should be noted that... The height is H. The width is W×ρ (i.e., 3). The number of channels is C / 2.

[0177] Similarly, to obtain Then, the vertical predictor can Sending to the vertical mixer, similarly, the horizontal predictor can also... Send to the vertical mixer. Then, the vertical mixer can also separately... as well as Perform similar processing to obtain the first globally expanded feature map in the vertical direction. (The aforementioned fifth feature map) and the second globally extended feature map in the vertical direction. (The aforementioned fifth feature map), The first global extended feature (the aforementioned fifth feature) is included in the 18 image patches selected vertically upwards. This includes the second global extended feature (the aforementioned fifth feature) contained in the 18 image patches selected vertically upwards. It should be noted that... The height is H×ρ (i.e., 3). The width is W. The number of channels is C / 2. The height is H×ρ (i.e., 3). The width is W. The number of channels is C / 2.

[0178] (3) After obtaining the fifth features of N image blocks, the dynamic image block unit can also stitch together the fifth features of M image blocks to obtain the sixth features of M image blocks. It should be noted that the height of the whole formed by the M image blocks is the same as the height of the whole formed by the sixth features of the M image blocks (i.e., the sixth feature map), and the width of the whole formed by the M image blocks is the same as the width of the whole formed by the sixth features of the M image blocks. However, for any one of the M image blocks, the number of channels of that image block is less than the number of channels of the sixth feature of that image block.

[0179] As in the example above, the horizontal mixer can also... as well as By stitching the images together, a stitched feature map is obtained in the horizontal direction. (The aforementioned sixth feature map) contains the stitching features of 18 image patches selected horizontally (the aforementioned sixth feature). It should be noted that... The height is H. The width is W×ρ. The number of channels is 2C. The vertical mixer can also... as well as By stitching the images together, we obtain a stitched feature map in the vertical direction. (The aforementioned sixth feature map) contains the stitching features of 18 image patches selected vertically upwards (the aforementioned sixth feature). It should be noted that... The height is H×ρ, The width is W. The number of channels is 2C.

[0180] (4) After obtaining the fifth features of N image blocks, the dynamic image block unit also performs a full connection on the sixth features of M image blocks to obtain the seventh features of M image blocks. Then, the seventh features of M image blocks are used as the fusion result of M image blocks. It should be noted that the height of the whole formed by the M image blocks is the same as the height of the whole formed by the seventh features of the M image blocks (i.e., the seventh feature map), and the width of the whole formed by the M image blocks is the same as the width of the whole formed by the seventh features of the M image blocks. For any image block among the M image blocks, the number of channels of the image block is the same as the number of channels of the seventh feature of the image block.

[0181] As in the example above, the horizontal mixer can also obtain the fused feature map in the horizontal direction using the following formula:

[0182]

[0183] In the above formula, The horizontally fused feature map (the aforementioned seventh feature map) contains the fused features (the aforementioned seventh feature) of 18 image patches selected horizontally. It should be noted that... The height is H. The width is W×ρ. The number of channels is C.

[0184] At the same time, the vertical mixer can also acquire fused feature maps in the vertical direction. (The aforementioned seventh feature map) contains the fusion features of 18 image patches selected vertically upwards (the aforementioned seventh feature). It should be noted that... The height is H×ρ, The width is W. The number of channels is C.

[0185] 505. Based on the fusion results of M image patches, obtain the processing results of the target image.

[0186] After obtaining the fusion result of M image patches, the target model can perform a series of processing on the fusion result of the M image patches to obtain the processing result of the target image. Then, the processing result of the target image can be used to complete the visual task.

[0187] Specifically, the target model can obtain the processing results of the target image in the following ways:

[0188] (1) The dynamic image block unit can also perform a second full connection on N image blocks to obtain the eighth feature of N image blocks. It should be noted that the height of the whole formed by the N image blocks is the same as the height of the whole formed by the eighth feature of the N image blocks (i.e., the eighth feature map), the width of the whole formed by the N image blocks is the same as the width of the whole formed by the eighth feature of the N image blocks, and for any one of the N image blocks, the number of channels of the image block is the same as the number of channels of the eighth feature of the image block.

[0189] (2) After obtaining the eighth features of N image blocks, the dynamic image block unit can also use preset weights to perform a weighted summation of the fusion result of M image blocks and the eighth features of M image blocks, thereby obtaining the ninth features of M image blocks. It should be noted that the height of the whole formed by the M image blocks is the same as the height of the whole formed by the eighth features of the M image blocks (i.e., a part of the eighth feature map), the width of the whole formed by the M image blocks is the same as the width of the whole formed by the eighth features of the M image blocks, and for any image block among the M image blocks, the number of channels of the image block is the same as the number of channels of the eighth feature of the image block.

[0190] (3) After obtaining the eighth features of N image blocks, the dynamic image block unit can also use preset weights to perform a weighted summation of the eighth features of the NM image blocks (excluding the M image blocks) to obtain the ninth features of the NM image blocks. It should be noted that the height of the whole formed by the NM image blocks is the same as the height of the whole formed by the eighth features of the NM image blocks (i.e., another part of the eighth feature map), the width of the whole formed by the NM image blocks is the same as the width of the whole formed by the eighth features of the NM image blocks, and for any image block among the NM image blocks, the number of channels of the image block is the same as the number of channels of the eighth feature of the image block.

[0191] (4) After obtaining the ninth features of N image blocks, the dynamic image block unit also sends the ninth features of the N image blocks to the processing unit so that the processing unit processes the ninth features of the N image blocks to obtain the processing result of the target image. It should be noted that the height of the whole formed by the N image blocks is the same as the height of the whole formed by the ninth features of the N image blocks (i.e., the ninth feature map), the width of the whole formed by the N image blocks is the same as the width of the whole formed by the eighth features of the N image blocks, and for any image block among the N image blocks, the number of channels of the image block is the same as the number of channels of the ninth feature of the image block.

[0192] For example, such as Figure 7 As shown ( Figure 7 This is another structural schematic diagram of the dynamic image block unit provided in an embodiment of this application. Figure 7 yes Figure 6 Based on the above, after obtaining the fusion features of the 18 image patches selected horizontally, these fusion features can be combined with the remaining 18 unselected image patches in the horizontal direction to form the first new feature map. Similarly, after obtaining the fusion features of the 18 image patches selected vertically, these fusion features can be combined with the remaining 18 unselected image patches in the vertical direction to form the second new feature map. The dynamic image patch unit can also perform a fully connected operation on the original 36 image patches to obtain a third new feature map (the aforementioned eighth feature map), which contains a certain feature of these 36 image patches (the aforementioned eighth feature). It should be noted that the height of these three new feature maps is H, the width is W, and the number of channels is C. Then, the dynamic image patch unit can perform a weighted sum of these three new feature maps to obtain the latest feature map (the aforementioned ninth feature map), which contains the latest feature of these 36 image patches (the aforementioned ninth feature).

[0193] The above is a detailed explanation of the target model for the first structure. The target model for the second structure will be introduced below. Figure 8As shown ( Figure 8 (This is another structural schematic diagram of the target model provided in the embodiment of this application). The target model includes a dynamic multilayer perceptron module, which includes a dynamic image block unit and a processing unit. The processing unit includes two normalization units, one channel unit, and two jump connection units.

[0194] Therefore, the first normalization unit can acquire N image blocks of the target image, normalize these N image blocks to obtain N new image blocks (the aforementioned normalized N image blocks), and then input these new N image blocks into the dynamic image block unit for various processing (see reference). Figure 5 The illustrated embodiment (not detailed here) yields the ninth feature of the new N image patches. Next, the first skip-connection unit superimposes the ninth feature of the new N image patches with the existing N image patches to obtain the tenth feature of the new N image patches. Following this, the second normalization unit normalizes the tenth feature of the new N image patches to obtain the eleventh feature of the new N image patches. Then, the channel unit aggregates the eleventh feature of the new N image patches along the channel dimension to obtain the twelfth feature of the new N image patches. Finally, the second skip-connection unit superimposes the twelfth feature of the new N image patches with the ninth feature of the new N image patches to obtain the processed result of the target image.

[0195] The above is a detailed explanation of the target model for the second structure. The target model for the third structure will be introduced below. Figure 9 As shown ( Figure 9 (This is another structural diagram of the target model provided in an embodiment of this application). The target model includes ten dynamic multilayer perceptron modules and four downsampling modules. These modules are connected in series, and the internal structure of each dynamic multilayer perceptron module can be referred to... Figure 8 The structure of the dynamic multilayer perceptron module is shown.

[0196] It should be understood that the embodiments of this application are only illustrative of three target models and do not limit the structural configuration of the target models provided in the embodiments of this application. For example, in Figure 8 In the target model shown, the number of normalization units and the number of skip units can be increased or decreased, etc. For example, in... Figure 9 In the target model shown, the number of dynamic multilayer perceptron modules and downsampling modules can be increased or decreased, etc. There are no restrictions here, and the settings can be made according to actual needs.

[0197] Furthermore, the target models provided in the embodiments of this application (including DynamicMLP-T, DynamicMLP-S, DynamicMLP-B, and DynamicMLP-L in Table 1) can be compared with models of a certain part of related technologies (including the remaining models in Table 1 excluding DynamicMLP-T, DynamicMLP-S, DynamicMLP-B, and DynamicMLP-L). The comparison results are shown in Table 1.

[0198] Table 1

[0199]

[0200]

[0201]

[0202] Furthermore, the target models provided in the embodiments of this application (including DynamicMLP-T, DynamicMLP-S, DynamicMLP-B, and DynamicMLP-L in Table 2) can be compared with models from another part of related technologies (including the remaining models in Table 2 excluding DynamicMLP-T, DynamicMLP-S, DynamicMLP-B, and DynamicMLP-L). The comparison results are shown in Table 2.

[0203] Table 2

[0204]

[0205]

[0206] As can be seen from Tables 1 and 2, the target model provided in this application embodiment performs better on visual tasks and requires less computation (FLOPs).

[0207] In this embodiment, after receiving N image blocks of the target image, the target model first evaluates the N image blocks to obtain evaluation values ​​for each block. Then, the target model uses these evaluation values ​​as selection criteria to select M image blocks from the N blocks. Next, the target model fuses the M image blocks to obtain a fused result. Finally, the target model performs a series of processing steps on the fused result to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image blocks indicate the importance of the content presented by each block, the M image blocks selected by the target model based on these evaluation values ​​are usually the more important parts of the target image. Therefore, in obtaining the processed result of the target image, the target model only performs fusion operations on these M image blocks, without fusing the remaining NM image blocks. This effectively reduces the computational load, shortens the overall image processing time, and improves the efficiency of image processing.

[0208] Furthermore, the target model provided in this application embodiment can automatically filter the original N image patches, retaining only M of them for fusion. Therefore, this novel model possesses automatic image patch selection functionality, eliminating the need to introduce additional neural network models. Consequently, no extra work is required during model training and application, saving various costs and resources.

[0209] The above is a detailed description of the image processing method provided in the embodiments of this application. The model training method provided in the embodiments of this application will be introduced below. Figure 10 A schematic flowchart of the model training method provided in the embodiments of this application is shown below. Figure 10 As shown, the method includes:

[0210] 1001. Input the target image into the model to be trained to obtain the processing result of the target image. The model to be trained is used to: obtain N image patches of the target image; evaluate the N image patches to obtain the evaluation values ​​of the N image patches, which are used to indicate the importance of the content presented by the N image patches; determine M image patches from the N image patches based on the evaluation values ​​of the N image patches, where N > M ≥ 2; fuse the M image patches to obtain the fusion result of the M image patches; and obtain the processing result of the target image based on the fusion result of the M image patches.

[0211] In this embodiment, when it is necessary to obtain a model with image processing capabilities, a model to be trained (i.e., an untrained neural network model) and a batch of training data can be obtained. This batch of training data includes the actual processing results of the target image, which can also be understood as the label of the target image, and this label is known.

[0212] After obtaining the target image, it can be divided into N image patches, which are then input into a training model. The training model processes these N image patches to obtain the processing result of the target image. Specifically, the training model is used to: acquire the N image patches of the target image; evaluate the N image patches to obtain evaluation values, which indicate the importance of the content presented by each patch; determine M image patches from the N image patches based on their evaluation values, where N > M ≥ 2; fuse the M image patches to obtain a fused result; and obtain the processing result of the target image based on the fused result.

[0213] In one possible implementation, the model to be trained is used to: perform a first fully connected operation on N image patches to obtain first features of the N image patches; pool the first features of the N image patches to obtain second features of the N image patches; multiply the first features of the N image patches and the second features of the N image patches to obtain third features of the N image patches, and use the third features of the N image patches as evaluation values ​​of the N image patches.

[0214] In one possible implementation, N image patches form an X-row, Y-column image patch array. The model to be trained is used to: select the P image patches with the largest evaluation values ​​in the i-th row of image patches, i = 1, ..., X, M = XP, P ≥ 1; or, select the K image patches with the largest evaluation values ​​in the j-th column of image patches, j = 1, ..., Y, M = YK, K ≥ 1.

[0215] In one possible implementation, the model to be trained is further configured to: perform a weighted summation of the evaluation values ​​of N image patches with the first features of the N image patches to obtain the fourth features of the N image patches; multiply the fourth features of the N image patches with the evaluation values ​​of M image patches to obtain the fifth features of the M image patches; the model to be trained is configured to: concatenate the M image patches with the fifth features of the M image patches to obtain the sixth features of the M image patches; perform a fully connected operation on the sixth features of the M image patches to obtain the seventh features of the M image patches, and the seventh features of the M image patches are used as the fusion result of the M image patches.

[0216] In one possible implementation, the model to be trained is used to: perform a second fully connected layer on N image patches to obtain the eighth features of the N image patches; perform a weighted summation of the fusion result of M image patches and the eighth features of the M image patches to obtain the ninth features of the M image patches; perform a weighted summation of the eighth features of the NM image patches other than the M image patches to obtain the ninth features of the NM image patches; and process the ninth features of the N image patches to obtain the processing result of the target image.

[0217] In one possible implementation, the processing includes at least one of the following: normalization, aggregation, or addition.

[0218] In one possible implementation, the model to be trained is also used to: normalize N image patches to obtain normalized N image patches.

[0219] 1002. Based on the processing results and the actual processing results of the target image, obtain the target loss.

[0220] After obtaining the processing result of the target image, since the actual processing result of the target image is known, a preset target loss function can be used to calculate the processing result of the target image and the actual processing result of the target image, thereby obtaining the target loss. The target loss can be used to indicate the difference between the processing result of the target image and the actual processing result of the target image.

[0221] 1003. Based on the target loss, update the parameters of the model to be trained until the model training conditions are met, and obtain the target model.

[0222] After obtaining the target loss, the model parameters of the model to be trained can be updated based on the target loss to obtain the updated model to be trained. Then, the next batch of training data is obtained, and the updated model to be trained is trained based on the next batch of training data (i.e., steps 1001 to 1003 are re-executed) until the model training conditions are met (e.g., the target loss converges, etc.), and the target model can be obtained.

[0223] The target model trained in this embodiment has image processing capabilities. Specifically, after receiving N image blocks of a target image, the target model first evaluates the N image blocks to obtain evaluation values ​​for each block. Then, the target model uses these evaluation values ​​as selection criteria to select M image blocks from the N blocks. Next, the target model fuses the M image blocks to obtain a fused result. Finally, the target model performs a series of processing steps on the fused result to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image blocks indicate the importance of the content presented by each block, the M image blocks selected by the target model based on these evaluation values ​​are usually the more important parts of the target image. Therefore, in obtaining the processed result of the target image, the target model only performs fusion operations on these M image blocks, without fusing the remaining NM image blocks. This effectively reduces the computational load, shortens the overall image processing time, and improves the efficiency of image processing.

[0224] The above is a detailed description of the image processing method and model training method provided in the embodiments of this application. The image processing device and model training device provided in the embodiments of this application will be described below. Figure 11 A schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application is shown below. Figure 11 As shown, the device includes a target model and comprises:

[0225] The first acquisition module 1101 is used to acquire N image blocks of the target image;

[0226] Evaluation module 1102 is used to evaluate N image patches and obtain evaluation values ​​for N image patches. The evaluation values ​​of N image patches are used to indicate the importance of the content presented by the N image patches.

[0227] The determination module 1103 is used to determine M image blocks from the N image blocks based on the evaluation values ​​of the N image blocks, where N > M ≥ 2;

[0228] The fusion module 1104 is used to fuse M image blocks to obtain the fusion result of the M image blocks;

[0229] The second acquisition module 1105 is used to acquire the processing result of the target image based on the fusion result of M image blocks.

[0230] In this embodiment, after receiving N image blocks of the target image, the target model first evaluates the N image blocks to obtain evaluation values ​​for each block. Then, the target model uses these evaluation values ​​as selection criteria to select M image blocks from the N blocks. Next, the target model fuses the M image blocks to obtain a fused result. Finally, the target model performs a series of processing steps on the fused result to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image blocks indicate the importance of the content presented by each block, the M image blocks selected by the target model based on these evaluation values ​​are usually the more important parts of the target image. Therefore, in obtaining the processed result of the target image, the target model only performs fusion operations on these M image blocks, without fusing the remaining NM image blocks. This effectively reduces the computational load, shortens the overall image processing time, and improves the efficiency of image processing.

[0231] In one possible implementation, the evaluation module 1102 is configured to: perform a first fully connected operation on N image blocks to obtain first features of the N image blocks; pool the first features of the N image blocks to obtain second features of the N image blocks; multiply the first features and second features of the N image blocks to obtain third features of the N image blocks, and use the third features of the N image blocks as the evaluation values ​​of the N image blocks.

[0232] In one possible implementation, N image blocks form an X-row, Y-column image block array. The determining module 1103 is used to: select the P image blocks with the largest evaluation values ​​in the i-th row of image blocks, i = 1, ..., X, M = XP, P ≥ 1; or select the K image blocks with the largest evaluation values ​​in the j-th column of image blocks, j = 1, ..., Y, M = YK, K ≥ 1.

[0233] In one possible implementation, the device further includes: a summation module for weighted summation of the evaluation values ​​of N image blocks with the first features of the N image blocks to obtain a fourth feature of the N image blocks; a multiplication module for multiplying the fourth features of the N image blocks with the evaluation values ​​of M image blocks to obtain a fifth feature of the M image blocks; and a fusion module 1104 for: concatenating the M image blocks with the fifth features of the M image blocks to obtain a sixth feature of the M image blocks; performing a full connection on the sixth features of the M image blocks to obtain a seventh feature of the M image blocks, wherein the seventh feature of the M image blocks is used as the fusion result of the M image blocks.

[0234] In one possible implementation, the second acquisition module 1105 is configured to: perform a second fully connected operation on N image blocks to obtain the eighth features of the N image blocks; perform a weighted summation of the fusion result of M image blocks and the eighth features of the M image blocks to obtain the ninth features of the M image blocks; perform a weighted summation of the eighth features of the NM image blocks other than the M image blocks to obtain the ninth features of the NM image blocks; and process the ninth features of the N image blocks to obtain the processing result of the target image.

[0235] In one possible implementation, the processing includes at least one of the following: normalization, aggregation, or addition.

[0236] In one possible implementation, the device further includes a normalization module for normalizing N image blocks to obtain normalized N image blocks.

[0237] Figure 12 A schematic diagram of the model training apparatus provided in the embodiments of this application is shown below. Figure 12 As shown, the device includes:

[0238] The input module 1201 is used to input the target image into the model to be trained to obtain the processing result of the target image. The model to be trained is used to: acquire N image patches of the target image; evaluate the N image patches to obtain evaluation values ​​of the N image patches, the evaluation values ​​of the N image patches being used to indicate the importance of the content presented by the N image patches; determine M image patches from the N image patches based on the evaluation values ​​of the N image patches, where N > M ≥ 2; fuse the M image patches to obtain the fusion result of the M image patches; and obtain the processing result of the target image based on the fusion result of the M image patches.

[0239] The acquisition module 1202 is used to acquire the target loss based on the processing result and the actual processing result of the target image;

[0240] The update module 1203 is used to update the parameters of the model to be trained based on the target loss until the model training conditions are met, thereby obtaining the target model.

[0241] The target model trained in this embodiment has image processing capabilities. Specifically, after receiving N image blocks of a target image, the target model first evaluates the N image blocks to obtain evaluation values ​​for each block. Then, the target model uses these evaluation values ​​as selection criteria to select M image blocks from the N blocks. Next, the target model fuses the M image blocks to obtain a fused result. Finally, the target model performs a series of processing steps on the fused result to obtain the processed result of the target image. In the aforementioned process, since the evaluation values ​​of the N image blocks indicate the importance of the content presented by each block, the M image blocks selected by the target model based on these evaluation values ​​are usually the more important parts of the target image. Therefore, in obtaining the processed result of the target image, the target model only performs fusion operations on these M image blocks, without fusing the remaining NM image blocks. This effectively reduces the computational load, shortens the overall image processing time, and improves the efficiency of image processing.

[0242] In one possible implementation, the model to be trained is used to: perform a first fully connected operation on N image patches to obtain first features of the N image patches; pool the first features of the N image patches to obtain second features of the N image patches; multiply the first features of the N image patches and the second features of the N image patches to obtain third features of the N image patches, and use the third features of the N image patches as evaluation values ​​of the N image patches.

[0243] In one possible implementation, N image patches form an X-row, Y-column image patch array. The model to be trained is used to: select the P image patches with the largest evaluation values ​​in the i-th row of image patches, i = 1, ..., X, M = XP, P ≥ 1; or, select the K image patches with the largest evaluation values ​​in the j-th column of image patches, j = 1, ..., Y, M = YK, K ≥ 1.

[0244] In one possible implementation, the model to be trained is further configured to: perform a weighted summation of the evaluation values ​​of N image patches with the first features of the N image patches to obtain the fourth features of the N image patches; multiply the fourth features of the N image patches with the evaluation values ​​of M image patches to obtain the fifth features of the M image patches; the model to be trained is configured to: concatenate the M image patches with the fifth features of the M image patches to obtain the sixth features of the M image patches; perform a fully connected operation on the sixth features of the M image patches to obtain the seventh features of the M image patches, and the seventh features of the M image patches are used as the fusion result of the M image patches.

[0245] In one possible implementation, the model to be trained is used to: perform a second fully connected layer on N image patches to obtain the eighth features of the N image patches; perform a weighted summation of the fusion result of M image patches and the eighth features of the M image patches to obtain the ninth features of the M image patches; perform a weighted summation of the eighth features of the NM image patches other than the M image patches to obtain the ninth features of the NM image patches; and process the ninth features of the N image patches to obtain the processing result of the target image.

[0246] In one possible implementation, the processing includes at least one of the following: normalization, aggregation, or addition.

[0247] In one possible implementation, the model to be trained is also used to: normalize N image patches to obtain normalized N image patches.

[0248] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0249] This application also relates to an execution device. Figure 13 This is a schematic diagram of the execution device provided in an embodiment of this application. Figure 13 As shown, the execution device 1300 can specifically manifest as a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited here. Among them, the execution device 1300 may deploy... Figure 11The image processing apparatus described in the corresponding embodiment is used to implement Figure 5 The corresponding embodiment describes the image processing function. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303, and a memory 1304 (wherein the execution device 1300 may have one or more processors 1303). Figure 13 (Taking a processor as an example), processor 1303 may include application processor 13031 and communication processor 13032. In some embodiments of this application, receiver 1301, transmitter 1302, processor 1303 and memory 1304 may be connected via bus or other means.

[0250] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0251] Processor 1303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0252] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1303. The processor 1303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1303 or by instructions in software form. The processor 1303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1303 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1304. Processor 1303 reads the information in memory 1304 and, in conjunction with its hardware, completes the steps of the above method.

[0253] Receiver 1301 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1302 can be used to output digital or character information through the first interface; transmitter 1302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1302 may also include a display device such as a display screen.

[0254] In one embodiment of this application, the processor 1303 is used to... Figure 5 The target image processing result is obtained using the target model in the corresponding embodiment.

[0255] This application also relates to a training device. Figure 14 This is a schematic diagram of the structure of a training device provided in an embodiment of this application. Figure 14As shown, the training device 1400 is implemented by one or more servers. The training device 1400 can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) 1414 (e.g., one or more processors) and memory 1432, and one or more storage media 1430 (e.g., one or more mass storage devices) for storing application programs 1442 or data 1444. The memory 1432 and storage media 1430 can be temporary or persistent storage. The program stored in the storage media 1430 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the training device. Furthermore, the CPU 1414 may be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the training device 1400.

[0256] The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458; or, one or more operating systems 1441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0257] Specifically, the training equipment can perform Figure 10 The model training method in the corresponding embodiment.

[0258] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0259] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.

[0260] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0261] For details, please refer to Figure 15 , Figure 15 This is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1500. The NPU 1500 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1503, which is controlled by the controller 1504 to extract matrix data from the memory and perform multiplication operations.

[0262] In some implementations, the arithmetic circuit 1503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1503 is a two-dimensional pulsating array. The arithmetic circuit 1503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1503 is a general-purpose matrix processor.

[0263] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1508.

[0264] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 via Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 via DMAC.

[0265] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1509.

[0266] The Bus Interface Unit (BIU) 1513 is used by the instruction fetch memory 1509 to fetch instructions from external memory, and also by the memory access controller 1505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0267] The DMAC is mainly used to move input data from external memory DDR to unified memory 1506, or to weight data to weight memory 1502, or to input data to input memory 1501.

[0268] The vector computation unit 1507 includes multiple processing units that further process the output of the computation circuit 1503 when necessary, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.

[0269] In some implementations, the vector computation unit 1507 can store the processed output vector in the unified memory 1506. For example, the vector computation unit 1507 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1503, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1507 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1503, for example, for use in subsequent layers of the neural network.

[0270] The instruction fetch buffer 1509 connected to the controller 1504 is used to store the instructions used by the controller 1504;

[0271] Unified memory 1506, input memory 1501, weighted memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0272] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0273] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0274] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0275] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0276] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An image processing method, characterized in that, The method is implemented through a target model, and the method includes: Obtain N image patches from the target image; The N image patches are evaluated to obtain evaluation values ​​for the N image patches, and the evaluation values ​​of the N image patches are used to indicate the importance of the content presented by the N image patches; Based on the evaluation values ​​of the N image blocks, M image blocks are determined from the N image blocks, where N > M ≥ 2; The evaluation values ​​of the N image blocks are weighted and summed with the first features of the N image blocks to obtain the fourth features of the N image blocks, wherein the first features of the N image blocks are obtained by performing a first fully connected operation on the N image blocks; The fourth features of the N image blocks and the evaluation values ​​of the M image blocks are multiplied together to obtain the fifth features of the M image blocks; The M image blocks are concatenated with the fifth feature of the M image blocks to obtain the sixth feature of the M image blocks; A full connection is performed on the sixth feature of the M image blocks to obtain the seventh feature of the M image blocks, and the seventh feature of the M image blocks is used as the fusion result of the M image blocks; Based on the fusion results of the M image blocks, the processing result of the target image is obtained.

2. The method according to claim 1, characterized in that, The evaluation of the N image patches to obtain the evaluation values ​​of the N image patches includes: Perform a first fully connected layer on the N image blocks to obtain the first feature of the N image blocks; Pooling is performed on the first features of the N image blocks to obtain the second features of the N image blocks; The first feature and the second feature of the N image blocks are multiplied together to obtain the third feature of the N image blocks, and the third feature of the N image blocks is used as the evaluation value of the N image blocks.

3. The method according to claim 2, characterized in that, The N image blocks form an X-row, Y-column image block array. The step of determining M image blocks from the N image blocks based on their evaluation values ​​includes: In the image patch of the i-th row, select the P image patches with the largest evaluation values, i=1,...,X,M=XP, P≥1, or... In the j-th column of the image blocks, select the K image blocks with the largest evaluation values, j=1,...,Y,M=YK,K≥1.

4. The method according to claim 3, characterized in that, The processing result of obtaining the target image based on the fusion result of the M image patches includes: A second fully connected layer is applied to the N image blocks to obtain the eighth feature of the N image blocks; The fusion result of the M image blocks and the eighth feature of the M image blocks are weighted and summed to obtain the ninth feature of the M image blocks; The ninth feature of the NM image blocks is obtained by weighted summing of the NM image blocks other than the M image blocks and the eighth feature of the NM image blocks; The ninth feature of the N image blocks is processed to obtain the processing result of the target image.

5. The method according to claim 4, characterized in that, The processing includes at least one of the following: normalization, aggregation, or addition.

6. The method according to any one of claims 1 to 5, characterized in that, Before evaluating the N image patches to obtain the evaluation values ​​for the N image patches, the method further includes: The N image blocks are normalized to obtain N normalized image blocks.

7. A model training method, characterized in that, The method includes: The target image is input into the model to be trained to obtain the processing result of the target image. The model to be trained is used to: acquire N image patches of the target image; evaluate the N image patches to obtain evaluation values ​​of the N image patches, the evaluation values ​​of the N image patches are used to indicate the importance of the content presented by the N image patches; based on the evaluation values ​​of the N image patches, determine M image patches from the N image patches, where N > M ≥ 2; and perform a weighted summation of the evaluation values ​​of the N image patches with the first features of the N image patches to obtain the fourth features of the N image patches, wherein the evaluation values ​​of the N image patches are... The first feature is obtained by performing a first fully connected operation on the N image patches; the fourth feature of the N image patches and the evaluation values ​​of the M image patches are multiplied together to obtain the fifth feature of the M image patches; the M image patches are concatenated with the fifth feature of the M image patches to obtain the sixth feature of the M image patches; the sixth feature of the M image patches is fully connected to obtain the seventh feature of the M image patches, and the seventh feature of the M image patches is used as the fusion result of the M image patches; based on the fusion result of the M image patches, the processing result of the target image is obtained; Based on the processing results and the actual processing results of the target image, the target loss is obtained; Based on the target loss, the parameters of the model to be trained are updated until the model training conditions are met, and the target model is obtained.

8. The method according to claim 7, characterized in that, The model to be trained is used for: Perform a first fully connected layer on the N image blocks to obtain the first feature of the N image blocks; Pooling is performed on the first features of the N image blocks to obtain the second features of the N image blocks; The first feature and the second feature of the N image blocks are multiplied together to obtain the third feature of the N image blocks, and the third feature of the N image blocks is used as the evaluation value of the N image blocks.

9. The method according to claim 8, characterized in that, The N image patches form an X-row, Y-column image patch array, and the model to be trained is used for: In the image patch of the i-th row, select the P image patches with the largest evaluation values, i=1,...,X,M=XP, P≥1, or... In the j-th column of the image blocks, select the K image blocks with the largest evaluation values, j=1,...,Y,M=YK,K≥1.

10. The method according to claim 9, characterized in that, The model to be trained is used for: A second fully connected layer is applied to the N image blocks to obtain the eighth feature of the N image blocks; The fusion result of the M image blocks and the eighth feature of the M image blocks are weighted and summed to obtain the ninth feature of the M image blocks; The ninth feature of the NM image blocks is obtained by weighted summing of the NM image blocks other than the M image blocks and the eighth feature of the NM image blocks; The ninth feature of the N image blocks is processed to obtain the processing result of the target image.

11. The method according to claim 10, characterized in that, The processing includes at least one of the following: normalization, aggregation, or addition.

12. The method according to any one of claims 7 to 11, characterized in that, The model to be trained is also used for: The N image blocks are normalized to obtain N normalized image blocks.

13. An image processing apparatus, characterized in that, The device includes a target model, and the device comprises: The first acquisition module is used to acquire N image blocks of the target image; An evaluation module is used to evaluate the N image blocks and obtain evaluation values ​​for the N image blocks. The evaluation values ​​of the N image blocks are used to indicate the importance of the content presented by the N image blocks. The determination module is used to determine M image blocks from the N image blocks based on the evaluation values ​​of the N image blocks, where N > M ≥ 2; The summation module is used to perform a weighted summation of the evaluation values ​​of the N image blocks and the first features of the N image blocks to obtain the fourth features of the N image blocks, wherein the first features of the N image blocks are obtained by performing a first fully connected operation on the N image blocks; The multiplication module is used to multiply the fourth features of the N image blocks and the evaluation values ​​of the M image blocks to obtain the fifth features of the M image blocks; The fusion module is used to concatenate the M image blocks with the fifth features of the M image blocks to obtain the sixth features of the M image blocks; and to perform a full connection on the sixth features of the M image blocks to obtain the seventh features of the M image blocks, with the seventh features of the M image blocks serving as the fusion result of the M image blocks. The second acquisition module is used to acquire the processing result of the target image based on the fusion result of the M image blocks.

14. A model training device, characterized in that, The device includes: An input module is used to input the target image into the model to be trained to obtain the processing result of the target image. The model to be trained is used to: acquire N image patches of the target image; evaluate the N image patches to obtain evaluation values ​​for the N image patches, the evaluation values ​​of the N image patches indicating the importance of the content presented by the N image patches; based on the evaluation values ​​of the N image patches, determine M image patches from the N image patches, where N > M ≥ 2; and perform a weighted summation of the evaluation values ​​of the N image patches with the first features of the N image patches to obtain a fourth feature of the N image patches, wherein the N... The first feature of the image patch is obtained by performing a first fully connected operation on the N image patches; the fourth feature of the N image patches is multiplied by the evaluation values ​​of the M image patches to obtain the fifth feature of the M image patches; the M image patches are concatenated with the fifth feature of the M image patches to obtain the sixth feature of the M image patches; the sixth feature of the M image patches is fully connected to obtain the seventh feature of the M image patches, and the seventh feature of the M image patches is used as the fusion result of the M image patches; based on the fusion result of the M image patches, the processing result of the target image is obtained; The acquisition module is used to acquire the target loss based on the processing result and the actual processing result of the target image; The update module is used to update the parameters of the model to be trained based on the target loss until the model training conditions are met, thereby obtaining the target model.

15. An image processing apparatus, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the image processing device performs the method as described in any one of claims 1 to 12.

16. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 12.

17. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 12.

Images