Image processing method, device and storage medium

By segmenting and clustering high-resolution images, the problems of increased resource consumption and complexity caused by larger resolution in existing technologies are solved, achieving efficient image processing results.

CN118521472BActive Publication Date: 2024-11-08ZHEJIANG DAHUA TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410996953.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-08
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

Existing technologies require adjustments to the resolution of large multimodal models when processing high-resolution images, resulting in high training resource consumption and increased time and space complexity, making it difficult to efficiently process high-resolution image features.

Method used

By dividing the image to be processed into blocks, extracting global and local image features separately, and clustering the stitched features to generate clustered image features to perform the target image processing task, the increase in the number of words caused by the increase in resolution is avoided.

Benefits of technology

It takes into account both global and fine-grained feature modeling, reduces the time and space complexity of the algorithm, improves the processing accuracy of high-resolution images, and reduces the computational resource requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118521472B_ABST
    Figure CN118521472B_ABST
Patent Text Reader

Abstract

The present application discloses an image processing method, device and storage medium, the image processing method comprising: obtaining an image to be processed and a plurality of image blocks obtained by dividing the image to be processed, extracting image features from the image to be processed and the plurality of image blocks respectively, obtaining global image features and local image features; splicing the global image features and each local image feature to obtain spliced ​​image features; clustering the word units contained in the spliced ​​image features, obtaining clustered image features based on the representative word units corresponding to each cluster cluster obtained by clustering; performing the target image processing task using the clustered image features to obtain the image processing results. The method can take into account both modeling global features and fine-grained local features, and reduce the time complexity and space complexity of the algorithm by clustering the word units contained in the spliced ​​image features, so as to effectively process large-resolution images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] In the field of artificial intelligence, large multi-modal models (LMMs) have become one of the frontier hotspots of research and application. Large multi-modal models refer to large deep learning models that can simultaneously process, understand, and integrate multiple types of data (such as text, images, voice, video, etc.). These models usually have a large parameter scale (up to billions or even tens of billions) and a highly complex neural network structure to achieve efficient modeling and accurate reasoning of different modal information.

[0003] In image processing technology, LMMs can be used to complete image processing tasks such as target detection, image retrieval, scene recognition, etc. However, when the image resolution is large, such as the image frame obtained by video acquisition of the target scene, the resolution is generally large. In order to obtain fine-grained visual features, the input resolution of the feature encoder needs to be increased.

[0004] At present, most image processing methods reuse pre-trained encoders such as CLIP (Contrastive Language-Image Pre-Training), OpenCLIP, Eva-CLIP, etc., and their input resolution is very small (such as 224x224). This is because changes in resolution often require retraining of the model, which consumes a lot of training resources. In addition, larger resolution leads to larger extracted image features, which increases the time and space complexity of image processing. Summary of the invention

[0005] In order to solve the above technical problems, the present application at least provides an image processing method, device and storage medium.

[0006] The first aspect of the present application provides an image processing method, which includes: obtaining an image to be processed and multiple image blocks obtained by dividing the image to be processed, extracting image features from the image to be processed and the multiple image blocks respectively, and obtaining global image features corresponding to the image to be processed and local image features corresponding to each image block; wherein the global image features and the local image features are both composed of multiple word units; splicing the global image features and each local image feature to obtain a spliced ​​image feature; clustering the word units contained in the spliced ​​image features, and obtaining clustered image features based on representative word units corresponding to each cluster cluster obtained by clustering; and using the clustered image features to execute a target image processing task to obtain an image processing result.

[0007] In one embodiment, image feature extraction is performed on an image to be processed and multiple image blocks respectively to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block respectively, including: performing scale change processing on the image to be processed and each image block respectively to obtain an image to be processed and an image block of a preset scale; performing image encoding processing on the image to be processed and the image block of the preset scale respectively to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block respectively.

[0008] In one embodiment, word units contained in the spliced ​​image features are clustered, and clustered image features are obtained based on representative word units corresponding to each cluster cluster obtained by clustering, including: multi-stage clustering of the word units contained in the spliced ​​image features, and initial clustering features corresponding to each stage are obtained based on representative word units corresponding to each cluster cluster obtained by clustering in each stage; wherein the number of cluster clusters generated in each stage is different; and the initial clustering features corresponding to each stage are spliced ​​to obtain clustered image features.

[0009] In one embodiment, multi-stage clustering is performed on the words contained in the spliced ​​image features, and initial clustering features corresponding to each stage are obtained based on representative words corresponding to each cluster cluster obtained by clustering in each stage, including: multi-stage clustering is performed on the words contained in the spliced ​​image features using a preset clustering algorithm to obtain clustering results corresponding to each stage; wherein the clustering results contain multiple clusters; the words in each cluster cluster are fused separately to obtain representative words corresponding to each cluster cluster; for any stage, all representative words corresponding to the stage are combined to obtain initial clustering features corresponding to the stage.

[0010] In one embodiment, a preset clustering algorithm is used to perform multi-stage clustering on word units contained in the spliced ​​image features to obtain clustering results corresponding to each stage, including: if the current stage is not the first stage, the initial clustering features obtained in the previous stage are used as features to be clustered; if the current stage is the first stage, the spliced ​​image features are used as features to be clustered; the preset clustering algorithm is used to cluster the word units contained in the features to be clustered to obtain the clustering results corresponding to the current stage.

[0011] In one embodiment, an image processing model is pre-trained, and the image processing model includes a feature encoding network, a feature mapping network and a prediction network; image features are extracted for the image to be processed and each image block respectively, to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block respectively, including: inputting the image to be processed and multiple image blocks into the feature encoding network respectively, to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block respectively output by the feature encoding network; using clustered image features to perform a target image processing task to obtain an image processing result, including: inputting the clustered image features into the feature mapping network to obtain an image embedding output by the feature mapping network; inputting the image embedding into the prediction network to obtain an image processing result output by the prediction network.

[0012] In one embodiment, the prediction network is constructed based on a large language model; the method also includes: obtaining prompt text content corresponding to the target image processing task; performing text feature extraction on the prompt text content to obtain text prompt embedding; inputting the image embedding into the prediction network to obtain the image processing result output by the prediction network, including: inputting the image embedding and the text prompt embedding into the prediction network to obtain the image processing result output by the prediction network.

[0013] In one embodiment, a low-rank matrix is ​​added to the prediction network; the method also includes: obtaining a sample image, the sample image corresponds to a sample label; inputting the sample image into a pre-trained image processing model to obtain an actual output result output by the pre-trained image processing model; using the actual output result and the sample label, calculating the model training loss; adjusting the network parameters of the feature mapping network and the low-rank matrix based on the model training loss, and not adjusting the network parameters of the feature encoding network and the prediction network.

[0014] According to a second aspect of the present application, there is provided an image processing device, which includes: a feature extraction module, which is used to obtain an image to be processed and multiple image blocks obtained by dividing the image to be processed, and to extract image features from the image to be processed and the multiple image blocks, respectively, to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block; wherein the global image features and the local image features are both composed of multiple word units; a feature splicing module, which is used to splice the global image features and each local image feature to obtain spliced ​​image features; a feature clustering module, which is used to cluster the word units contained in the spliced ​​image features, and to obtain clustered image features based on representative word units corresponding to each cluster cluster obtained by clustering; and an image processing module, which is used to perform a target image processing task using the clustered image features to obtain an image processing result.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned image processing method.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned image processing method when the program instructions are executed by a processor.

[0017] The above scheme obtains an image to be processed and multiple image blocks obtained by dividing the image to be processed, extracts image features from the image to be processed and the multiple image blocks respectively, obtains global image features corresponding to the image to be processed and local image features corresponding to each image block; splices the global image features and each local image feature to obtain spliced ​​image features; clusters the words contained in the spliced ​​image features, and obtains clustered image features based on representative words corresponding to each cluster cluster obtained by clustering; uses the clustered image features to perform the target image processing task to obtain image processing results, which can take into account modeling global features and fine-grained local features, and by clustering the words contained in the spliced ​​image features, it avoids increasing the time complexity and space complexity of the algorithm due to an increase in the number of words caused by an increase in resolution, thereby improving the image processing accuracy of large-resolution images and reducing the computing resource requirements for processing large-resolution images.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart of an image processing method shown in an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram showing image block division of an image to be processed according to an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of image feature extraction shown in an exemplary embodiment of the present application;

[0024] Figure 5 is a schematic diagram of multi-stage clustering shown in an exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of performing image processing based on an image processing model, shown in an exemplary embodiment of the present application;

[0026] Figure 7 is a block diagram of an image processing device shown in an exemplary embodiment of the present application;

[0027] Figure 8 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0028] Fig. 9 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0029] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0030] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0031] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0032] The image processing method provided in the embodiment of the present application is described below.

[0033] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected to each other in communication.

[0034] The number of the terminal 110 may be one or more. The terminal 110 may be a camera, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto.

[0035] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0036] In one example, the server 120 may obtain an image to be processed from the terminal 110, perform a target image processing task on the image to be processed, and obtain an image processing result, wherein the target image processing task includes but is not limited to target detection, event recognition, image retrieval, and the like, and the image processing result may be a target detection result, an event recognition result, an image retrieval result, etc. Then, the server 120 may store the image processing result locally, transmit it back to the terminal 110, or transmit it to other terminals 110.

[0037] In one example, a client running a target application is installed in the terminal 110, such as an application that provides image processing functions. The server 120 may be a background server of the target application, which is used to provide background services for the client of the target application, so that the terminal 110 performs image processing on the image to be processed and obtains an image processing result.

[0038] It should be noted that all tasks related to feature extraction of images can use the image processing method provided by this application, and an example of a target image processing task based on multimodal data is given below:

[0039] Image retrieval tasks: retrieval between text and image, retrieval between text and video, retrieval between image and image, etc. For example, on e-commerce platforms, you can search for related product images through text descriptions entered by the terminal; on news or social media platforms, return related video content through text queries entered by the terminal; or in the fields of copyright protection, image recognition, etc., by comparing the matching degree between images, return images that match the input image.

[0040] Visual question answering and assisted reading: By understanding the information combined with pictures and texts, and answering related questions. For example, in scenarios such as smart e-books and barrier-free reading tools, questions are input through the terminal, the picture and text information is analyzed, and the corresponding answers are output.

[0041] Intelligent creation and editing: For example, intelligent painting tools can generate image works based on the text description input by the terminal; the automatic subtitle generation system can automatically add accurate subtitles to the video; the intelligent editing software can automatically filter and edit video materials according to the text script, etc.

[0042] Autonomous driving and robotics technology: Autonomous driving vehicles and robots need to process data from multiple sensors such as camera images, lidar, microphones, etc., and combine the above sensor data to understand complex road environments, the behavior of target objects, voice commands, etc., to help vehicles or robots identify obstacles, understand traffic signs, predict the behavior of other road users, and achieve safe and efficient autonomous navigation.

[0043] The above examples are only examples of target image processing tasks in some application scenarios, and this application does not limit the target image processing tasks.

[0044] In the image fusion method provided in the embodiment of the present application, the execution subject of each step can be the terminal 110, such as the client of the target application installed and running in the terminal 110, or the server 120, or the terminal 110 and the server 120 can interact and cooperate to execute, that is, part of the steps of the method are executed by the terminal 110 and the other steps are executed by the server 120.

[0045] See also Figure 2 , Figure 2 is a flowchart of an image processing method shown in an exemplary embodiment of the present application. The image processing method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0046] like Figure 2 As shown, the image processing method at least includes steps S210 to S240, which are described in detail as follows:

[0047] Step S210: Obtain an image to be processed and multiple image blocks obtained by dividing the image to be processed, extract image features from the image to be processed and the multiple image blocks respectively, and obtain global image features corresponding to the image to be processed and local image features corresponding to each image block; wherein both the global image features and the local image features are composed of multiple word units.

[0048] Obtain an image to be processed, which may be a single image captured by an image acquisition device, or any video frame in a video stream captured by a video acquisition device. In addition, the image to be processed may be a color image, a grayscale image, a depth image, etc., which is not limited in the present application.

[0049] The image to be processed is divided into multiple image blocks.

[0050] Exemplarily, a block division parameter corresponding to the image to be processed is obtained, and the block division parameter is used to indicate a specific way of dividing the image to be processed into image blocks.

[0051] For example, see Figure 3 , Figure 3 FIG. 2 shows a schematic diagram of dividing an image to be processed into image blocks according to an exemplary embodiment. Figure 3 As shown, in the implementable method 1, the block division parameter indicates that the image to be processed is divided in equal proportion, and the size of each image block finally obtained is consistent, such as dividing the image to be processed into 2*2 and 3*3 image blocks in equal proportion; in the implementable method 2, the block division parameter indicates that the image to be processed is divided in non-equal proportion, and the size of each image block finally obtained is inconsistent, for example, the image block division size is determined according to the importance of different image contents in the image to be processed, and the image block division is performed according to the image block division size. For example, the image to be processed is split into foreground and background, a smaller image block division size is set for the foreground image content, and a larger image block division size is set for the background image content.

[0052] It should be noted that the specific implementation method of dividing the image to be processed into blocks can be flexibly selected according to the actual application scenario, and this application does not limit this.

[0053] Optionally, image blocks may overlap with each other, that is, the same image content may be included in multiple image blocks at the same time. For example, the overlap rate of the image content in each image block is set based on the importance of the image content, so that the important image content is included in multiple image blocks, such as first dividing the image to be processed into equal proportions to obtain multiple equal proportion image blocks, and then dividing the important image content area (such as the foreground area) in the image to be processed into image blocks again to obtain overlapping image blocks, so that the important image content area can be included not only in the equal proportion image blocks, but also in the overlapping image blocks obtained by dividing the important image content area separately, thereby improving the processing accuracy of the important image content.

[0054] Then, image features are extracted from the image to be processed and the multiple image blocks respectively, to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block.

[0055] Exemplarily, the image features include, but are not limited to, one or more of color features, spatial features, edge features, depth features, and the like.

[0056] Specifically, image features are extracted from the image to be processed and the image block to realize image coding processing of the image to be processed and the image block, and global image features and multiple local image features are obtained. The global image features and local image features are composed of multiple tokens. Among them, encoders such as CLIP (Contrastive Language-Image Pre-Training), OpenCLIP, and Eva-CLIP can be used for image feature extraction.

[0057] Among them, token is the encoded representation of image patch. The image or image block to be processed is divided into multiple patches of fixed size. Each patch is regarded as a token and converted into a high-dimensional vector through linear mapping, position encoding, etc., so that the original pixel matrix is ​​converted into a token sequence that can be understood by the computer.

[0058] For example, the image to be processed is divided into 3*3 image blocks, and image features are extracted from the image to be processed and each image block to obtain a set of global image features and local image features. , any image feature There are N tokens, and the feature vector dimension of each token is D. Therefore, the encoding resolution can be increased by 9 times by image segmentation, while taking into account both global features and fine-grained local features.

[0059] Step S220: splicing the global image feature and each local image feature to obtain a spliced ​​image feature.

[0060] The global image feature and each local image feature are concatenated to obtain a concatenated image feature, which contains all word units of the global image feature and all word units of the local image feature.

[0061] For example, the global image feature and the local image feature set , any image feature There are N tokens, and the feature vector dimension of each token is D. Flatten and concatenate the global image features and local image features to obtain the concatenated image features , the spliced ​​image features contain 10N tokens, and the feature vector dimension of each token is D.

[0062] Step S230: clustering the word-grams contained in the spliced ​​image features, and obtaining clustered image features based on representative word-grams corresponding to each cluster obtained by clustering.

[0063] The tokens contained in the spliced ​​image features are used as the data set to be clustered, so as to cluster the tokens in the data set, and obtain the clustered image features according to the clustering results.

[0064] Among them, k-means clustering algorithm (k-means clustering algorithm), fuzzy clustering mean (FCM) algorithm, etc. can be used to cluster tokens, and this application does not limit this.

[0065] Exemplarily, multiple clusters are obtained by clustering the spliced ​​image features, each cluster contains similar tokens, and the cluster center corresponding to the cluster can be used as the representative word (representative token) corresponding to the cluster; all tokens contained in the cluster can also be fused, and the fused token can be used as the representative token corresponding to the cluster; any token in the cluster can also be selected as the representative token corresponding to the cluster, which is not limited in this application. Then, according to the representative token corresponding to each cluster, the cluster image feature is obtained.

[0066] Optionally, the implementation method for determining the representative word can be flexibly selected according to actual conditions. For example, a representative word determination strategy can be selected based on the number of tokens contained in the clusters, the minimum similarity between the tokens contained in the clusters, etc., and the representative word corresponding to each cluster can be determined according to the representative word determination strategy.

[0067] Exemplarily, multi-stage clustering may also be performed on the spliced ​​image features to obtain clustered image features according to the multi-stage clustering results.

[0068] For example, the first stage clustering is performed on the word units contained in the spliced ​​image features, and the first intermediate features are obtained based on the representative word units corresponding to each cluster cluster obtained by the first stage clustering; the second stage clustering is performed on the word units contained in the first intermediate features, and the second intermediate features are obtained based on the representative word units corresponding to each cluster cluster obtained by the second stage clustering. The second intermediate features can be used as clustered image features, and the splicing result of the first intermediate features and the second intermediate features can also be used as clustered image features.

[0069] For another example, a first-stage clustering is performed on the words contained in the spliced ​​image features, and a first intermediate feature is obtained based on a representative word corresponding to each cluster cluster obtained by the first-stage clustering; a second-stage clustering is performed on the words contained in the spliced ​​image features, and a second intermediate feature is obtained based on a representative word corresponding to each cluster cluster obtained by the second-stage clustering, wherein the number of cluster clusters obtained by the first-stage clustering and the second-stage clustering are different; and the splicing result of the first intermediate feature and the second intermediate feature is used as the clustered image feature.

[0070] Optionally, the number of clusters (or the number of clusters in each stage of clustering) can be determined based on the resolution of the image to be processed, the important image content content (such as foreground ratio) of the image to be processed, etc. For example, the higher the resolution of the image to be processed and the greater the important image content content, the greater the number of clusters. An upper limit on the number of clusters (or clusters in each stage of clustering) can be set to avoid excessive clustering image features affecting the efficiency of subsequent processing.

[0071] Step S240: Utilize the clustered image features to perform the target image processing task and obtain an image processing result.

[0072] Among them, target image processing tasks include but are not limited to target detection tasks, event recognition tasks, image retrieval tasks, etc.

[0073] The target image processing task is performed according to the clustered image features to obtain the image processing result.

[0074] For example, clustered image features are input into a pre-trained prediction network to obtain image processing results output by the prediction network.

[0075] For another example, the image processing method of the present application is implemented based on large multi-modal models (Large Multi-modal Models, LMMs). The clustered image features and the embedded expressions corresponding to other models (such as text, language, etc.) can be input into the prediction network corresponding to the LMMs to obtain the image processing results output by the prediction network.

[0076] For example, for an image retrieval task, image features of candidate images are pre-stored, and the vector distance between the clustered image features and the image features of each candidate image can be calculated, and the candidate image with the smallest vector distance is selected as the target image corresponding to the image retrieval task to obtain the image processing result.

[0077] The present application can take into account both global features and fine-grained local features in modeling by extracting the image features of the image to be processed and the image features of the image blocks respectively, and by clustering the words contained in the spliced ​​image features, it avoids the increase in the time complexity and space complexity of the algorithm due to the increase in the number of words caused by the increase in resolution, thereby improving the image processing accuracy of large-resolution images and reducing the computing resource requirements for processing large-resolution images.

[0078] Next, some embodiments of the present application are described by way of example.

[0079] In some embodiments, in step S210, image feature extraction is performed on the image to be processed and the multiple image blocks respectively to obtain the global image feature corresponding to the image to be processed and the local image feature corresponding to each image block, including:

[0080] Step S211: performing scale change processing on the image to be processed and each image block respectively to obtain the image to be processed and the image block of preset scale.

[0081] Step S212: performing image coding processing on the image to be processed and the image block of a preset scale respectively, to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block.

[0082] The scale change processing is used to enlarge or reduce the size of the image to be processed and each image block to a preset scale, such as changing the scale of the image to be processed and each image block to 224*224 respectively.

[0083] Then, image coding processing is performed on the image to be processed and the image block of a preset scale to obtain the global image features corresponding to the image to be processed and the local image features corresponding to each image block.

[0084] For an example, see Figure 4 , Figure 4 This is a schematic diagram of image feature extraction shown in an exemplary embodiment of the present application, as shown in Figure 4 As shown, the image to be processed and each image block are scaled to obtain the image to be processed and the image block of the preset scale. Then, the image to be processed and the image block of the preset scale are respectively input into the feature encoding network for image feature extraction. The image features of the image to be processed output by the feature encoding network are used as global image features, and the image features of the image blocks output by the feature encoding network are used as local image features, such as obtaining local image features 1, local image features 2...local image features n, where n is the number of image blocks.

[0085] Although the resolution of image encoding can be improved by extracting global image features and local image features separately, the increase in the number of tokens will seriously affect the reasoning efficiency if subsequent reasoning is performed directly. Therefore, the present application further clusters the spliced ​​image features corresponding to the global image features and local image features.

[0086] In some implementations, clustering the word units contained in the spliced ​​image features in step S230, and obtaining clustered image features based on representative word units corresponding to each cluster obtained by clustering, includes:

[0087] Step S231: multi-stage clustering is performed on the word units contained in the spliced ​​image features, and based on the representative word units corresponding to each cluster cluster obtained by clustering in each stage, the initial cluster features corresponding to each stage are obtained; wherein the number of cluster clusters generated in each stage is different.

[0088] Exemplarily, a preset clustering algorithm is used to perform multi-stage clustering on the words contained in the spliced ​​image features to obtain clustering results corresponding to each stage; wherein the clustering results contain multiple clustering clusters; the words in each clustering cluster are fused respectively to obtain representative words corresponding to each clustering cluster; for any stage, all representative words corresponding to the stage are combined to obtain the initial clustering features corresponding to the stage.

[0089] For example, the preset clustering algorithm is the k-means clustering algorithm, where k refers to the number of cluster centers. K words are randomly selected as the initial cluster centers, and then the distance between each word and each cluster center is calculated, and each word is assigned to the cluster center closest to it. The cluster centers and the words assigned to them represent a cluster. Each time a word is assigned, the cluster center of the cluster is recalculated based on the existing words in the cluster. The above process will be repeated until a termination condition is met. The termination condition can be that no (or a minimum number of) words are reassigned to different clusters, no (or a minimum number of) cluster centers change again, the error sum of squares is locally minimized, etc.

[0090] Multi-stage clustering means performing multiple clustering processes, and the number of clusters generated in each stage is different.

[0091] Exemplarily, if the current stage is not the first stage, the initial clustering features obtained in the previous stage are used as the features to be clustered; if the current stage is the first stage, the spliced ​​image features are used as the features to be clustered; a preset clustering algorithm is used to cluster the words contained in the features to be clustered to obtain the clustering results corresponding to the current stage.

[0092] Of course, the spliced ​​image features may be used as features to be clustered in each stage, and a specific multi-stage clustering method may be selected according to a specific application scenario, which is not limited in this application.

[0093] Step S232: concatenate the initial cluster features corresponding to each stage to obtain cluster image features.

[0094] For an example, see Figure 5 , Figure 5 A schematic diagram of multi-stage clustering is shown in an exemplary embodiment of the present application. Figure 5As shown, the spliced ​​image features corresponding to the image to be processed are obtained. The k value of the first stage is set to 128, and the k-means clustering algorithm is used to cluster the tokens of the spliced ​​image features to obtain 128 cluster clusters corresponding to the first stage. The word units in each cluster cluster are fused to obtain the representative word units corresponding to each cluster cluster. Combined with all the representative word units corresponding to the first stage, the initial clustering feature P1 corresponding to the first stage is obtained; then, the k value of the second stage is set to 64, and the k-means clustering algorithm is used to cluster the tokens of P1. Similarly, the initial clustering feature P2 corresponding to the second stage is obtained; the k value of the third stage is set to 32, and the k-means clustering algorithm is used to cluster the tokens of P2. Similarly, the initial clustering feature P3 corresponding to the third stage is obtained. Then, P1, P2, and P3 are spliced ​​to obtain multi-scale clustered image features.

[0095] In the above method, multi-scale visual features can be obtained through multi-stage clustering, thereby improving the accuracy and information expression ability of clustered image features.

[0096] Feature clustering can reduce the number of tokens and improve the efficiency of subsequent reasoning. In the clustering process, the representation of the same image content in global image features and in local image features are considered at the same time, which can take into account both global information and fine-grained local information.

[0097] In some embodiments, an image processing model is pre-trained, and the image processing model includes a feature encoding network, a feature mapping network and a prediction network; image features are extracted for the image to be processed and each image block, respectively, to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block, including: inputting the image to be processed and multiple image blocks into the feature encoding network, respectively, to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block output by the feature encoding network; using clustered image features to perform a target image processing task to obtain an image processing result, including: inputting the clustered image features into the feature mapping network, to obtain an image embedding output by the feature mapping network; inputting the image embedding into the prediction network, to obtain an image processing result output by the prediction network.

[0098] Among them, the image processing model can be constructed based on large multi-modal models (Large Multi-modal Models, LMMs). Of course, the image processing model can also adopt other model architectures, which is not limited in this application.

[0099] Exemplarily, the prediction network is constructed based on a large language model; the method also includes: obtaining prompt text content corresponding to the target image processing task; performing text feature extraction on the prompt text content to obtain text prompt embedding; inputting the image embedding into the prediction network to obtain the image processing result output by the prediction network, including: inputting the image embedding and the text prompt embedding into the prediction network to obtain the image processing result output by the prediction network.

[0100] That is, the image processing model is a multimodal large model (MM-LMMs) based on a large language model, wherein the prediction network of the image processing model is constructed based on a large language model (LLMs), which reduces the amount of computation required to pre-train and align the features of each modality from scratch, and a multimodal large model (MM-LMMs) based on a large language model emerges. This type of method extracts modal features with the help of a pre-trained multimodal encoder, and uses the modal features as a language prompt to input them into LLMs. With the help of LLMs, multimodal feature alignment is completed, which can achieve more natural and efficient human-computer interaction, such as visual question-answering and dialogue.

[0101] In addition, the feature encoding network can be constructed based on CLIP (Contrastive Language-Image Pre-Training), OpenCLIP, Eva-CLIP, etc., and the feature mapping network can be constructed based on a multi-layer perceptron (MLP).

[0102] The prompt text content may be input by the terminal, such as obtaining the prompt text content according to the text, voice, etc. input by the terminal, for example, the prompt text content may be “what is the posture of the cat in the image”.

[0103] Then, the image embedding and text prompt embedding are input into the prediction network to obtain the image processing result output by the prediction network.

[0104] For an example, see Figure 6 , Figure 6 FIG. 1 is a schematic diagram of performing image processing based on an image processing model according to an exemplary embodiment of the present application. Figure 6As shown, the image to be processed and the n image blocks obtained by dividing the image to be processed are input into the feature encoding network to obtain the spliced ​​image features obtained by splicing the global image features and each local image feature, and then the spliced ​​image features are clustered to obtain clustered image features. The clustered image features are input into the feature mapping network to obtain the image embedding output by the feature mapping network, and the image embedding and the text prompt embedding are spliced ​​according to the instruction template and input into the large language model to obtain the image processing result output by the large language model. Among them, the feature dimensions of the image embedding and the text prompt embedding are consistent.

[0105] When training an image processing model, the image processing model is trained according to a sample image carrying a sample label, and the actual output result of the pre-trained image processing model is obtained by inputting the sample image into the pre-trained image processing model; the model training loss is calculated using the actual output result and the sample label; a LoRA branch is set, and the LoRA branch adds a low-rank matrix to the large language model, and the network parameters of the feature mapping network and the low-rank matrix are adjusted based on the model training loss, and the network parameters of the feature encoding network and the prediction network are not adjusted.

[0106] Since both the feature encoding network and the large language model can reuse the model parameters that have been trained in the prior art, that is, during the training process, the feature encoding network and the prediction network are frozen and their parameters are not updated. Only the network parameters of the training feature mapping network and the low-rank matrix are updated. The amount of training parameters is extremely low and the training efficiency is high.

[0107] The image processing method provided by the present application obtains an image to be processed and multiple image blocks obtained by dividing the image to be processed, extracts image features from the image to be processed and the multiple image blocks respectively, obtains global image features corresponding to the image to be processed and local image features corresponding to each image block; splices the global image features and each local image feature to obtain spliced ​​image features; clusters the words contained in the spliced ​​image features, and obtains clustered image features based on representative words corresponding to each cluster cluster obtained by clustering; uses the clustered image features to execute a target image processing task to obtain an image processing result, which can take into account both modeling global features and fine-grained local features, and by clustering the words contained in the spliced ​​image features, it is avoided that the time complexity and space complexity of the algorithm increase due to the increase in the number of words caused by the increase in resolution, thereby improving the image processing accuracy of large-resolution images and reducing the computing resource requirements for processing large-resolution images.

[0108] Figure 7 FIG. 1 is a block diagram of an image processing device shown in an exemplary embodiment of the present application. Figure 7As shown, the exemplary image processing device 700 includes: a feature extraction module 710, a feature splicing module 720, a feature clustering module 730 and an image processing module 740. Specifically:

[0109] The feature extraction module 710 is used to obtain the image to be processed and multiple image blocks obtained by dividing the image to be processed, and perform image feature extraction on the image to be processed and the multiple image blocks respectively to obtain the global image feature corresponding to the image to be processed and the local image feature corresponding to each image block; wherein the global image feature and the local image feature are both composed of multiple word units;

[0110] A feature stitching module 720 is used to stitch the global image feature and each local image feature to obtain a stitched image feature;

[0111] The feature clustering module 730 is used to cluster the word units contained in the spliced ​​image features, and obtain clustered image features based on the representative word units corresponding to each cluster obtained by clustering;

[0112] The image processing module 740 is used to perform a target image processing task using clustered image features to obtain an image processing result.

[0113] It should be noted that the image processing device provided in the above embodiment and the image processing method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the image processing device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0114] See also Figure 8 , Figure 8 800 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 800 includes a memory 801 and a processor 802, and the processor 802 is used to execute program instructions stored in the memory 801 to implement the steps in any of the above-mentioned image processing method embodiments. In a specific implementation scenario, the electronic device 800 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 800 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0115] Specifically, the processor 802 is used to control itself and the memory 801 to implement the steps in any of the above-mentioned image processing method embodiments. The processor 802 can also be called a central processing unit (CPU). The processor 802 may be an integrated circuit chip with signal processing capabilities. The processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 802 can be implemented by an integrated circuit chip.

[0116] See also Fig. 9 , Fig. 9 The computer-readable storage medium 900 stores program instructions 910 that can be executed by a processor, and the program instructions 910 are used to implement the steps in any of the above-mentioned image processing method embodiments.

[0117] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0118] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0119] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0120] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed and a plurality of image blocks obtained by dividing the image to be processed, extract image features from the image to be processed and the plurality of image blocks respectively, and obtain a global image feature corresponding to the image to be processed and a local image feature corresponding to each image block; wherein the global image feature and the local image feature are both composed of a plurality of word units; Splicing the global image feature and each local image feature to obtain a spliced ​​image feature; and determining the number of clusters in each stage according to the resolution of the image to be processed and the important image content of the image to be processed; Performing multi-stage clustering on the word units contained in the spliced ​​image features according to the number of clustering clusters in each stage, to obtain clustering results corresponding to each stage; wherein the clustering results contain the number of clustering clusters, and the number of clustering clusters corresponding to the clustering results generated in each stage is different; Merging the word units in each cluster respectively to obtain the representative word units corresponding to each cluster; For any stage, all representative word units corresponding to the stage are combined to obtain the initial clustering features corresponding to the stage; The initial clustering features corresponding to each stage are concatenated to obtain clustering image features; The clustered image features are used to perform a target image processing task to obtain an image processing result.

2. The method according to claim 1, characterized in that The extracting image features from the image to be processed and the multiple image blocks respectively to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block respectively includes: Performing scale change processing on the image to be processed and each image block respectively to obtain the image to be processed and the image block of preset scale; Image coding processing is performed on the image to be processed and the image block of the preset scale respectively to obtain the global image feature corresponding to the image to be processed and the local image feature corresponding to each image block.

3. The method according to claim 1, characterized in that The multi-stage clustering of the word units contained in the spliced ​​image features according to the number of clustering clusters in each stage to obtain the clustering results corresponding to each stage respectively includes: If the current stage is not the first stage, the initial clustering features obtained in the previous stage are used as the features to be clustered; if the current stage is the first stage, the spliced ​​image features are used as the features to be clustered; A preset clustering algorithm is used to cluster the word units contained in the features to be clustered to obtain a clustering result corresponding to the current stage.

4. The method according to any one of claims 1 to 3, characterized in that: An image processing model is pre-trained, wherein the image processing model includes a feature encoding network, a feature mapping network, and a prediction network; The extracting image features from the image to be processed and the multiple image blocks respectively to obtain global image features corresponding to the image to be processed and local image features corresponding to each image block respectively includes: Inputting the image to be processed and the multiple image blocks into the feature encoding network respectively, and obtaining the global image features corresponding to the image to be processed and the local image features corresponding to each image block output by the feature encoding network; The method of performing a target image processing task using the clustered image features to obtain an image processing result includes: Inputting the clustered image features into the feature mapping network to obtain an image embedding output by the feature mapping network; The image is embedded and input into the prediction network to obtain the image processing result output by the prediction network.

5. The method according to claim 4, characterized in that The prediction network is constructed based on a large language model; the method further comprises: Obtaining prompt text content corresponding to the target image processing task; Extracting text features from the prompt text content to obtain text prompt embedding; The embedding the image into the prediction network to obtain the image processing result output by the prediction network includes: The image embedding and the text prompt embedding are input into the prediction network to obtain the image processing result output by the prediction network.

6. The method according to claim 4, characterized in that The prediction network is added with a low-rank matrix; the method further comprises: Acquire a sample image, wherein the sample image corresponds to a sample label; Inputting the sample image into a pre-trained image processing model to obtain an actual output result output by the pre-trained image processing model; Calculating the model training loss using the actual output result and the sample label; The network parameters of the feature mapping network and the low-rank matrix are adjusted based on the model training loss, and the network parameters of the feature encoding network and the prediction network are not adjusted.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, and the processor is used to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Image processing method, image processing device, electronic equipment and storage medium

    CN110335199A

  • Target re-identification method, terminal equipment and computer readable storage medium

    CN114419408A

  • Model generation method and device, feature extraction method and device, equipment, medium and product

    CN114724148A

  • Image processing method and terminal equipment

    CN115294150A

  • Multi-camera cooperative monitoring method, device and equipment and storage medium

    CN117354471A