Image processing method and device, computer equipment and computer program product

Abstract

The invention is suitable for the technical field of image processing, and provides an image processing method and device, computer equipment and a computer program product, and the method comprises the steps: segmenting a high-resolution image to obtain a plurality of sub-images, and carrying out the size adjustment of the high-resolution image to obtain a full information image of the high-resolution image; obtaining visual features based on the plurality of sub-graphs and the total information graph; obtaining text features associated with the high-resolution image; and inputting the visual features and the text features into a multi-modal large model to obtain a model recognition result of the high-resolution image. Therefore, the recognition accuracy of the model on the high-resolution image content can be remarkably improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image processing method, apparatus, computer equipment and computer program product. Background Art

[0002] At present, in the design of smart cockpits, models are usually used to improve the intelligence of smart cockpit systems and the user's interactive experience. These models use sensors in the smart cockpit to collect data (such as high-resolution images) and process the high-resolution images. Among them, the commonly used processing scheme is to cut the high-resolution image, extract the local features in the high-resolution image, and obtain the model recognition result based on the features, so as to realize the recognition of passenger identity, gestures and surrounding objects, and fully perceive the environment and user behavior in the cockpit. However, the existing scheme directly processes the original high-resolution image, which may cause the loss of important information due to downsampling or cropping, and cause the model to deviate when understanding the image. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, computer device and computer program product, which can solve the technical problems that the existing methods directly process the original high-resolution images, which may cause the loss of important information due to downsampling or cropping, and may cause deviations in the model when understanding the image.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0005] Slicing the high-resolution image to obtain a plurality of sub-images, and resizing the high-resolution image to obtain a full information image of the high-resolution image;

[0006] Based on the plurality of sub-graphs and the full information graph, obtaining visual features;

[0007] obtaining text features associated with the high-resolution image;

[0008] The visual features and the text features are input into a multimodal large model to obtain a model recognition result of the high-resolution image.

[0009] In a possible implementation manner of the first aspect, segmenting the high-resolution image to obtain a plurality of sub-images, and resizing the high-resolution image to obtain a full information graph of the high-resolution image includes:

[0010] Determining a cutting strategy according to the length and width of the high-resolution image and the length and width of the predefined grid;

[0011] Slicing the high-resolution image based on the slice strategy to obtain a plurality of sub-images;

[0012] The high-resolution image is resized based on an input resolution of a visual encoder to generate a full information map; wherein the visual encoder is used to encode features of the full information map and the sub-map.

[0013] In a possible implementation manner of the first aspect, determining the image cutting strategy according to the length and width of the high-resolution image and the length and width of the predefined grid includes:

[0014] Determine a first aspect ratio according to the length and width of the high-resolution image, and determine a second aspect ratio according to the length and width of the predefined grid;

[0015] Obtaining a minimum value of a difference between the first aspect ratio and the second aspect ratio;

[0016] If there is only one minimum value, determining the image cutting strategy based on the length and width of the predefined grid corresponding to the minimum value;

[0017] If there are at least two minimum values, the image cutting strategy is determined according to the length and width of the high-resolution image, the length and width of the predefined grid corresponding to each minimum value, and the resolution of the sub-image.

[0018] In a possible implementation manner of the first aspect, segmenting the high-resolution image based on the image segmentation strategy to obtain the multiple sub-images includes:

[0019] Adjusting the resolution of the high-resolution image to an integer multiple of the input resolution of the visual encoder;

[0020] Based on the image cutting strategy, the high-resolution image after adjusting the resolution is cut into pieces to obtain a plurality of sub-images.

[0021] In a possible implementation manner of the first aspect, obtaining a visual feature based on the multiple sub-graphs and the full information graph includes:

[0022] Performing feature extraction on the full information graph to obtain global features;

[0023] Performing feature extraction on the plurality of sub-graphs to obtain local features;

[0024] Inputting the global feature and the local feature into a visual encoder to obtain the visual feature;

[0025] The resolution of the full information graph, the resolution of the sub-graph and the input resolution of the visual encoder are the same.

[0026] In a possible implementation manner of the first aspect, inputting the global feature and the local feature into a visual encoder to obtain the visual feature includes:

[0027] Arranging the local features and setting row labels based on the order of segmentation of the plurality of sub-graphs;

[0028] Flattening the global features and the local features after setting the row mark;

[0029] The flattened global features are used as the query vector of the cross attention, and the flattened local features are used as the key vector and the value vector to obtain the visual features.

[0030] In a possible implementation manner of the first aspect, inputting the visual features and the text features into a multimodal large model to obtain a model recognition result of the high-resolution image includes:

[0031] Mapping the visual features to input dimensions of a large language model;

[0032] The mapped visual features and the text features are input into the multimodal large model, and the model recognition results are output.

[0033] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0034] A segmentation module is used to segment a high-resolution image into multiple sub-images;

[0035] A size adjustment module, used for adjusting the size of the high-resolution image to obtain a full information map of the high-resolution image;

[0036] A visual feature generation module, used for obtaining visual features based on the plurality of sub-graphs and the full information graph;

[0037] A text feature acquisition module, used to acquire text features associated with the high-resolution image;

[0038] The determination module is used to input the visual features and the text features into a multimodal large model to obtain a model recognition result of the high-resolution image.

[0039] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image processing method described in any one of the first aspects above when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method described in any one of the first aspects above is implemented.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer device, the computer device executes the image processing method described in any one of the above-mentioned first aspects.

[0042] Compared with the prior art, the embodiments of the present application have the following beneficial effects: visual features are obtained through sub-graphs and full-information graphs, the sub-graphs retain local details of high-resolution images, and the full-information graphs retain global and contextual information of high-resolution images, and their combination can improve the understanding of image content, especially in complex scenes; in addition, visual features and text features are input into a large multimodal model together to obtain a model recognition result of the high-resolution image, and in-depth feature fusion and learning are performed by combining visual features and text features, which can perform richer multimodal learning and improve the model's recognition accuracy of image content.

[0043] In summary, by slicing subgraphs and generating full information graphs, as well as multimodal learning with text features, the model's recognition accuracy for high-resolution image content can be significantly improved.

[0044] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application;

[0047] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0048] Figure 3 It is a flowchart of the detailed steps of S201 in one embodiment of the present application;

[0049] Figure 4 It is a flowchart of the detailed steps of S202 in one embodiment of the present application;

[0050] Figure 5 is a flowchart of an image processing method provided by another embodiment of the present application;

[0051] Figure 6 is a schematic diagram of an application scenario of an image processing method provided by another embodiment of the present application;

[0052] Figure 7 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0053] Figure 8 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0055] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0056] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0057] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0058] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0059] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0060] In the design of smart cockpits, it is crucial to improve the intelligence of the system and the user's interactive experience. To this end, current solutions usually use a combination of a high-resolution image acquisition device 101 and an image processing device 102 to fully perceive the environment and user behavior in the cockpit. The image processing method provided in this application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, it includes: a high-resolution image acquisition device 101 and an image processing device 102 connected to the high-resolution image acquisition device 101.

[0061] The high-resolution image acquisition device 101 is used to acquire high-resolution images and to provide the acquired high-resolution images to the image processing device 102. The image processing device 102 can run a preset computer program to analyze and process the acquired high-resolution images, thereby outputting a model recognition result.

[0062] Optionally, the high-resolution image acquisition device 101 may be a camera installed on the vehicle. The camera may include a fisheye lens, a panoramic camera, a night vision camera, a thermal imaging camera, a high dynamic range camera, a depth camera, etc., and is not limited to the above cameras. Based on sensors such as the high-resolution image acquisition device 101, the situation in the cabin can be fully perceived from different angles and conditions. The image processing device 102 can collect information on the identity, gestures and surrounding objects of the passengers in real time, thereby improving the interactive experience of the smart cabin.

[0063] Optionally, the image processing device 102 can be deployed on a terminal device, such as a vehicle-mounted system, to achieve rapid response and meet application scenarios with high real-time requirements.

[0064] In the embodiment of the present application, the high-resolution image acquisition device 101 and the image processing device 102 can be deployed in the same device. In this case, the high-resolution image acquisition device 101 and the image processing device 102 can be regarded as different modules of the device. For example, the high-resolution image acquisition device 101 and the image processing device 102 are both modules in a terminal device (such as a vehicle), but the high-resolution image acquisition device 101 is deployed on the vehicle, and the image processing device 102 can be the control system or other related systems of the vehicle.

[0065] As mentioned above, existing solutions directly process the original high-resolution images, which may cause the loss of important information due to downsampling or cropping, and cause the model to deviate when understanding the image. In addition, existing solutions are mainly limited to target detection, have single functions, and fail to effectively combine multimodal large models, and cannot fully understand and respond to users' complex instructions and behaviors, further limiting the intelligent development of smart cockpit systems.

[0066] To solve the above problems, this application can significantly improve the model's recognition accuracy of high-resolution image content by slicing subgraphs and generating full information graphs, as well as multimodal learning with text features.

[0067] Figure 2 A schematic flowchart of an image processing method provided in the present application is shown. As an example but not a limitation, the method can be executed by a terminal device, which includes the above-mentioned image processing device 102, or the method can be executed by the image processing device 102 in the terminal device.

[0068] S201, dividing the high-resolution image into multiple sub-images, and resizing the high-resolution image to obtain a full information image of the high-resolution image.

[0069] The high-resolution image refers to video data or image data captured by a camera installed on the vehicle. For a description of the camera, see Figure 1 The illustrated embodiments will not be described in detail here.

[0070] Among them, a high-resolution image generally refers to an image with a higher pixel density, which means that the image contains a larger number of pixels, thereby being able to provide clearer and more delicate details. For example, common high-resolution standards include 1920x1080 (full HD) or 3840x2160 (4K). Optionally, the resolution of the high-resolution image is higher than the input resolution of the visual encoder in the image processing device 102 (see below for related content). Because the input images are mostly high-resolution images and the resolutions are inconsistent, the image processing device 102 cannot process various high-resolution images. Therefore, it is necessary to segment the high-resolution images to obtain sub-images that meet the input resolution requirements of the visual encoder in the image processing device 102.

[0071] The high-resolution image is divided into a uniform grid, and the image is divided into multiple sub-images of the same size.

[0072] The full information graph refers to an image that retains the global and contextual information of the high-resolution image. Optionally, the resolution of the full information graph also meets the input resolution requirement of the visual encoder in the image processing device 102.

[0073] Optionally, resizing the high-resolution image is resizing the high-resolution image in pixels. All sizes mentioned herein may be expressed as pixel sizes.

[0074] Among them, the high-resolution image is resized to generate a full information map. Common methods include but are not limited to the following methods:

[0075] Bicubic interpolation based resizing: The value of the resized pixel is calculated by weighted average of the surrounding 16 pixels.

[0076] Lanczos interpolation based resizing: uses a weighted function to consider multiple neighboring pixels for resizing.

[0077] Fourier transform-based resizing: By converting the image to the frequency domain for resizing and then inversely transforming it back to the spatial domain, the loss of details and blurring during image reduction can be effectively avoided.

[0078] Convolutional Neural Network-based Resizing: Use trained convolutional neural networks to resize high-resolution images to generate full-information graphs.

[0079] Among them, the resizing in the embodiment of the present application refers to resizing the high-resolution image so that the resized image not only meets the input resolution requirement of the visual encoder in the image processing device 102, but also retains the global and contextual information of the high-resolution image.

[0080] S202, obtaining visual features based on the multiple sub-graphs and the full information graph.

[0081] In an embodiment of the present application, features are extracted from the segmented sub-graphs and the full information graph, and combined to obtain visual features.

[0082] Among them, edge detection feature extraction algorithm, color histogram feature extraction algorithm, local feature extraction algorithm, feature extraction algorithm based on deep learning, etc. can be used to extract features from sub-graphs and full information graphs respectively. Afterwards, the extracted features are fused to enrich the feature expression.

[0083] Through the above technical solution, visual features can be effectively extracted from sub-graphs and full information graphs, thereby supporting subsequent intelligent analysis, recognition and interactive applications.

[0084] S203: Acquire text features associated with the high-resolution image.

[0085] In the embodiment of the present application, the text feature can be a text feature obtained based on a question input by the user, or a text feature obtained based on a pre-set question. For example, in an image recognition task, if a high-resolution image contains a book, the text associated with it can be a question asked by the user, such as "Who is the author of this book", or a preset question, such as a description of the subject of the book. After obtaining the text, the text is converted into a vector form so that the multimodal large model can process it.

[0086] S204: Input the visual features and the text features into a multimodal large model to obtain a model recognition result of the high-resolution image.

[0087] Among them, a multimodal big model can refer to a big model that can process and understand multiple types of data (i.e., multimodal data). It is no longer limited to a specific data set and can process multiple types of data such as text, images, audio, and video at the same time.

[0088] Specifically, the multimodal large model receives a large amount of multi-type data from different fields during training. Through cross-modal integration and learning of these large amounts of multi-type data, it learns rich semantic information from them, and exhibits new features and new functions that are beyond what a single modal combination can achieve. It has strong emergence and generalization capabilities, provides more accurate and reliable judgments, and overcomes the limitations of traditional deep learning methods in complex environments.

[0089] Optionally, a multimodal large model is deployed in the image processing device 102, and the image processing device 102 can input the visual features and the text features into the multimodal large model to obtain a model recognition result.

[0090] Optionally, the multimodal large model is deployed in other devices, and accordingly, S204 can be implemented in the following manner: the image processing device 102 can send the visual features and the text features to other devices, so that the other devices input the visual features and the text features into the multimodal large model to obtain a model recognition result. Then, the other devices send the model recognition result to the image processing device 102.

[0091] In the embodiment of the present application, in multimodal learning, the combination of visual features and text features can provide richer information to help the multimodal large model better understand and answer questions about the image. By analyzing the scene text in the image and combining it with relevant text information, the accuracy of image recognition can be improved.

[0092] As in the above example, the high-resolution image contains a book, and the user may be interested in the title of the book in the image. By extracting text features associated with this image, such as the question asked by the user, "Who is the author of this book?", natural language processing techniques can be used to convert this question into a text feature vector. These text feature vectors can then be input into a multimodal learning model together with the visual features extracted from the image, and the model can recognize the text in the image and give the title of the book as an answer. In this way, the model that combines image and text information can provide more accurate recognition results.

[0093] In an embodiment of the present application, a high-resolution image is segmented to obtain a plurality of sub-graphs, and the high-resolution image is resized to obtain a full information graph of the high-resolution image; visual features are obtained based on the plurality of sub-graphs and the full information graph; text features associated with the high-resolution image are obtained; and the visual features and the text features are input into a large multimodal model to obtain a model recognition result of the high-resolution image. Among them, the visual features are obtained through sub-graphs and full information graphs, the sub-graphs retain the local details of the high-resolution image, and the full information graph retains the global and contextual information of the high-resolution image, and their combination can improve the understanding of the image content, especially in complex scenes; in addition, the visual features and text features are input into a large multimodal model together to obtain the model recognition result of the high-resolution image, and the visual features and text features are combined to perform in-depth feature fusion and learning, which can perform richer multimodal learning and improve the model's recognition accuracy of image content.

[0094] In the prior art, in practical applications, the resolution and aspect ratio of the input images are often inconsistent, and the granularity of different information to be identified in the image is different, which may cause the model to lose key fine-grained visual information when processing the image, especially in high-resolution image processing. This affects the accuracy of model recognition and reduces the overall performance and user experience of the smart cockpit system. In order to solve the above problems, in one possible implementation, the high-resolution image is optimized by dynamically adjusting the image resolution and segmentation method to ensure the effective extraction of key features. At the same time, the high-resolution features are cleverly integrated into the full information graph, thereby reducing the number of visual features while retaining important information and improving computational efficiency. The specific implementation method is as follows: Figure 3 , Figure 3 The detailed process of S201 is shown, which includes dividing the high-resolution image into multiple sub-images, and resizing the high-resolution image to obtain a full information graph of the high-resolution image.

[0095] S2011, determining a cutting strategy according to the length and width of the high-resolution image and the length and width of the predefined grid.

[0096] The length and width of a high-resolution image can also be referred to as the size of a high-resolution image. The image width indicates the number of horizontal pixels in the image, that is, the width of the image from left to right, expressed in pixels. The image length indicates the number of vertical pixels in the image, that is, the length of the image from top to bottom, expressed in pixels.

[0097] Among them, multiple grids and the size of each grid are predefined. The grid width represents the number of horizontal pixels of each sub-image (grid), that is, the width of a single sub-image after segmentation, expressed in pixels. The grid length represents the number of vertical pixels of each sub-image (grid), that is, the length of a single sub-image after segmentation, expressed in pixels.

[0098] Here, we assume that the length and width of the high-resolution image are (H×W), and its resolution is also called (H×W), such as 1920x1080. 1920x1080 means that the width of the image is 1920 pixels and the length is 1080 pixels. Given the input resolution of the visual encoder is (a×a), (a×a) refers to the input resolution of a square, and a is the number of pixels of the side length of the input image. Set the maximum number of sub-images N c , define a set of grids [(1,1), (1,2), (2,1), ..., (m,n)], m×n <= N c. In (m,n), m represents the length of the grid, and n represents the width of the grid. (1,1) represents a square grid with a length and width of 1, (1,2) represents a rectangular grid with a length of 1 and a width of 2, and (2,1) represents a rectangular grid with a length of 2 and a width of 1. Assuming that the grid is defined as (1,2), it is a rectangular grid with a length of 1*a pixels and a width of 2*a pixels.

[0099] The image cutting strategy is a strategy for cutting a high-resolution image into a number of sub-images in a certain number of rows and columns.

[0100] In a possible implementation, S2011 specifically includes:

[0101] a. Determine a first aspect ratio according to the length and width of the high-resolution image, and determine a second aspect ratio according to the length and width of the predefined grid.

[0102] The ratio of the length H to the width W of the high-resolution image is used as the first aspect ratio. The width p of the predefined grid is w and length p h The ratio of is used as the second aspect ratio. There is only one first aspect ratio, but there may be multiple second aspect ratios. The number of predefined grids corresponds to the number of aspect ratios.

[0103] b. Obtain a minimum value of a difference between the first aspect ratio and the second aspect ratio.

[0104] The minimum value of the difference between the first aspect ratio and the second aspect ratio is determined based on the following formula.

[0105]

[0106] c, if there is only one minimum value X min , then based on the minimum value X min The length and width of the corresponding predefined grid determine the cutting strategy. The goal is to find the grid that minimizes this difference, so as to select the best grid. The length and width corresponding to the best grid are determined as the cutting strategy.

[0107] Among them, if there is only one minimum value X calculated by formula 1 min , then directly according to the minimum value X min Determine the corresponding second aspect ratio Then directly determine the width p w and length p h . Then the cutting strategy is p h ×p w . That is, split the high-resolution image into p h Line and pw List.

[0108] d, if there are at least two minimum values ​​X min , then according to the length and width of the high-resolution image, each minimum value X min The length and width of the corresponding predefined grid and the resolution of the sub-image determine the image cutting strategy.

[0109] If there are at least two minimum values ​​X min (i.e., at least two equal minimum values ​​are calculated), the graph cutting strategy is further determined according to the following formula.

[0110] min(|H * Wp h *a*p w *a|) (Formula 2)

[0111] Among them, when there are at least two minimum values ​​X min , assuming there are 3 minimum values, each minimum value X min All correspond to the width p w and length p h . Set each minimum value X min The corresponding width p w and length p h Substituting into formula 2, we get three values, namely |H*Wp h *a*p w *a|. The grid corresponding to the minimum value among the three values ​​is determined as the grid with the smallest difference, and then the width p of the grid with the smallest difference is determined. w and length p h . Then the cutting strategy is p h ×p w .

[0112] S2012: Slice the high-resolution image based on the slice strategy to obtain a plurality of sub-images.

[0113] Among them, as mentioned above, when determining the cutting strategy as p h ×p w , split the high-resolution image into p h Line and p w Column. Get p h ×p w Sub-pictures.

[0114] In a possible implementation, when determining the graph cutting strategy p h ×p w Finally, the resolution of the high-resolution image is adjusted to an integer multiple of the input resolution of the visual encoder [p h ×a,p w×a], which ensures that the high-resolution image can be segmented into sub-images suitable for the visual encoder input. Then, based on the segmentation strategy p h ×p w , segment the high-resolution image after adjusting the resolution to obtain p h ×p w sub-images to meet the input resolution requirement of the visual encoder.

[0115] S2013, resizing the high-resolution image based on the input resolution of the visual encoder to generate a full information map; wherein the visual encoder is used to encode features of the full information map and the sub-map.

[0116] Among them, the high-resolution image size is adjusted to the input resolution (a×a) of the visual encoder to generate a full information graph.

[0117] In the embodiment of the present application, in view of the problem of inconsistent resolution and aspect ratio of different input images in the intelligent cockpit system, and the image processing challenges brought by high-resolution images, the present application dynamically adjusts the resolution and cutting method of the image according to the principle that the grid should fit the aspect ratio of the input image, and the resolution loss is minimized, and the layout and number of image sub-images are adaptively adjusted to avoid resizing the sub-images to minimize the resolution loss while meeting the input requirements of the visual encoder. This method can maintain the original aspect ratio and detail information of the image as much as possible, while meeting the input requirements of the visual encoder, and avoiding the shortcomings of the traditional fixed cutting method in high-precision demand scenarios. In addition, in order to avoid the loss of global features, the present application also generates a full information map, so that the overall information and context information of the image can be retained while processing high-resolution images. This comprehensive processing method of global and local features further enhances the comprehensive understanding of image details, the relationship between objects in the image, and the global context, and provides richer and more accurate input data for the visual encoder, thereby achieving better performance in various visual tasks.

[0118] In one possible implementation, Figure 4 The detailed process of S202 in this application is shown.

[0119] S2021, extracting features from the full information graph to obtain global features.

[0120] S2022: Perform feature extraction on the plurality of sub-graphs to obtain local features.

[0121] The pre-trained visual encoder is used to extract features of the full information graph to obtain global features, and the pre-trained visual encoder is used to extract features of the sub-graph to obtain local features.

[0122] Among them, a visual encoder is deployed in the image processing device 102, and the image processing device 102 can obtain multiple sub-images and a full information image and input them into the visual encoder to obtain local features and global features.

[0123] S2023, input the global features and the local features into a visual encoder to obtain the visual features. The resolution of the full information graph, the resolution of the sub-graph and the input resolution of the visual encoder are the same.

[0124] In a possible implementation, S2023 specifically includes: arranging the local features and setting row labels based on the order of segmentation of the multiple sub-graphs; flattening the global features and the local features after setting row labels; using the flattened global features as the query vector of cross-attention, and using the flattened local features as the key vector and value vector to obtain the visual features.

[0125] Feature flattening is the process of converting a multidimensional data structure (such as a two-dimensional matrix or a three-dimensional tensor of an image) into a one-dimensional vector. Feature flattening helps reduce the number of network parameters and improve computational efficiency while retaining the key features of the input data. Figure 5 Examples in .

[0126] Among them, cross attention is an attention mechanism that allows a model to focus on how the information in one sequence is related to the information in another sequence when processing two or more different input sequences. For example, in multimodal learning tasks involving images and text, cross attention can help the model understand the connection between image content and related text.

[0127] In the embodiments of the present application, for ease of understanding, Figure 5 The image processing method is described by referring to the flowchart of FIG. Figure 5 As shown, the visual encoder outputs local features ( Figure 5 ), and process the local feature, including arranging and setting the row label ( Figure 5 , the 9 dark grey squares in three rows and three columns are the arranged features, and the light grey part is the row label). After that, the arranged and row labeled features are flattened, and the global features are also flattened. The flattened global features are used as the query vector Q of the cross attention, and the flattened local features are used as the key vector K and the value vector V to obtain the visual features.

[0128] For ease of understanding, here we combine Figure 5 For example. Assume that the visual encoder outputs local features ( Figure 5There are 9 dark gray squares in a row in the figure (A, B, C, D, E, F, G, H, I). After arranging the local features and setting the row labels, the features are represented as follows:

[0129] AB C1

[0130] DE F2

[0131] I3

[0132] After that, the features after arrangement and row labeling are flattened to obtain the following feature representation: A BC1D E F2G H I3. Here, the multidimensional features are converted to one dimension based on feature flattening, but it does not lose any information because all the features are sequentially contained in the one-dimensional vector "AB C1D E F2G H I3".

[0133] In an embodiment of the present application, in the processing of high-resolution images, in the face of the problem of a significant increase in the number of visual features, the present application proposes a feature injection strategy to optimize processing efficiency and maintain the key detail features of high-resolution images. The feature injection strategy includes multiple steps such as feature flattening and fusion, as well as residual connections, to meet the challenges brought by high-resolution images. In an embodiment of the present application, the high-resolution image is divided into multiple sub-graphs, and a feature representation is generated for each sub-graph. In order to maintain the spatial information of the image, it is necessary to restore the sub-graphs to their original arrangement and add learnable row labels after the features of each row of sub-graphs. These row labels serve as additional features to help enhance the spatial perception ability of the model and maintain the coherence between sub-graphs. In addition, the full information graph and sub-graph features are flattened, and the cross-attention mechanism is used for feature fusion. The full information graph features are used as query vectors, and the sub-graph features are used as key vectors and value vectors. This mechanism effectively combines global information with local details, and reduces the total number of features and reduces the computational complexity by compressing the fused features to the same number of features as the full information graph features, while retaining the key features of the high-resolution image. In addition, the introduction of residual connections adds the full information graph features to the fused visual features, which not only helps to retain the key information of high-resolution images, but also enhances the stability and accuracy of feature expression.

[0134] In a possible implementation, S204 specifically includes: mapping the visual features to the input dimensions of the large language model; inputting the mapped visual features and the text features into the multimodal large model, and outputting the model recognition results.

[0135] Optionally, the visual features are mapped to the input dimension of a large language model through a fully connected layer; the mapped visual features and the text features are input into the multimodal large model, and the model recognition results are output.

[0136] Among them, the fully connected layer is one of the basic components for building a neural network. It maps visual features to dimensions compatible with the large language model, and then inputs these mapped features together with text features into the multimodal large model, and finally outputs the recognition results of the model.

[0137] For ease of understanding, this is also combined with Figure 5 The flowchart of the image processing method shown in FIG. 4 is used to describe the multimodal large model. Generally, a fixed-length input vector is required. Therefore, Figure 5 As shown in the figure, the output dimension of the fully connected layer needs to match the input dimension of the multimodal large model. Therefore, the visual features need to be mapped to the input dimension of the large language model through the fully connected layer. Once the visual features are mapped to the correct dimension, they can be passed as input to the multimodal large model. While processing the visual features, the user's questions or related text information are also converted into text features. The visual features and text features are combined to form the input of the multimodal large model. The model will use these inputs to generate responses.

[0138] Among them, the combination of visual features and text features can be feature splicing.

[0139] Among them, the splicing methods may include: simple splicing, weighted splicing, attention splicing or gated splicing, etc.

[0140] Simple concatenation directly connects the features of different modalities to form a longer vector. In the embodiment of the present application, the above word vectors and visual features are directly connected in series.

[0141] Weighted concatenation assigns different weights to features of different modalities, and then performs weighted summation to form a new vector. The weights can be obtained through training and learning, or can be set based on experience. In the embodiment of the present application, different weights are assigned to the above-mentioned word vectors and visual features, and then weighted summation is performed to form a new vector.

[0142] Attention mechanism, use the attention mechanism to learn the relationship between different modal features, assign different weights according to the relationship, and then perform weighted summation. For example, the cross attention mechanism can be used to learn the correspondence between word vectors and visual features, and assign different weights according to the correspondence.

[0143] Gating mechanism: Use gating mechanism to select important information from different modal features and integrate them together. For example, you can use Gated Recurrent Unit (GRU) to learn the relationship between word vectors and visual features, and select important information for splicing based on the relationship.

[0144] It should be noted that this application only describes simple splicing, weighted splicing, attention splicing or gated splicing, which are only illustrative examples. This application may also adopt other splicing methods, and related splicing methods not described in the embodiments of this application are also within the scope of this application.

[0145] In the embodiment of the present application, dynamic high-resolution image processing is combined with a multimodal large model to achieve real-time interaction with users. The multimodal large model consists of a visual encoder, a mapping layer, and a generative large language model. The generative large language model can generate natural language responses based on the context without building a vector library, thereby improving the scalability and flexibility of the model.

[0146] In order to facilitate the understanding of the scheme, Figure 6 The application scenario diagram of the image processing method shown in the figure is used as an example. This example is only for illustration and not for limitation. Figure 6 As shown in the figure, a high-resolution image is input, and the high-resolution image is resized and segmented to obtain a full information graph and a sub-graph respectively. After that, the full information graph and the sub-graph are input into the visual encoder, compressed and fully connected to obtain visual features. At the same time, text features are obtained based on the word segmentation and vectorization of the text prompts. The text features and visual features are input into the large language model to output the answer "This person is holding a bottle in his hand. The bottle body is made of transparent plastic with a red label on it. The label says "Coca-Cola", which indicates that the bottle contains Coca-Cola beverages. In addition, there is a black cap on the top of the bottle. From these features, it can be determined that this is a bottle containing Coca-Cola."

[0147] It can be seen from the output information of this application scenario that based on the above-mentioned solution of this application, the recognition accuracy of the model for high-resolution image content can be significantly improved.

[0148] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0149] Corresponding to the image processing method described in the above embodiment, Figure 7 A structural block diagram of an image processing device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0150] Reference Figure 7 , the image processing device comprises:

[0151] A segmentation and resizing module, used to segment the high-resolution image to obtain a plurality of sub-images, and to resize the high-resolution image to obtain a full information image of the high-resolution image;

[0152] A visual feature generation module, used for obtaining visual features based on the plurality of sub-graphs and the full information graph;

[0153] A text feature acquisition module, used to acquire text features associated with the high-resolution image;

[0154] The output module is used to input the visual features and the text features into the multimodal large model to obtain the model recognition result of the high-resolution image.

[0155] In a possible implementation, the segmentation size adjustment module is used to:

[0156] Determining a cutting strategy according to the length and width of the high-resolution image and the length and width of the predefined grid;

[0157] Slicing the high-resolution image based on the slice strategy to obtain a plurality of sub-images;

[0158] The high-resolution image is resized based on an input resolution of a visual encoder to generate a full information map; wherein the visual encoder is used to encode features of the full information map and the sub-map.

[0159] In a possible implementation, the segmentation size adjustment module is used to:

[0160] Determine a first aspect ratio according to the length and width of the high-resolution image, and determine a second aspect ratio according to the length and width of the predefined grid;

[0161] Obtaining a minimum value of a difference between the first aspect ratio and the second aspect ratio;

[0162] If there is only one minimum value, determining the image cutting strategy based on the length and width of the predefined grid corresponding to the minimum value;

[0163] If there are at least two minimum values, the image cutting strategy is determined according to the length and width of the high-resolution image, the length and width of the predefined grid corresponding to each minimum value, and the resolution of the sub-image.

[0164] In a possible implementation, the segmentation size adjustment module is used to:

[0165] Adjusting the resolution of the high-resolution image to an integer multiple of the input resolution of the visual encoder;

[0166] Based on the image cutting strategy, the high-resolution image after adjusting the resolution is cut into pieces to obtain a plurality of sub-images.

[0167] In a possible implementation, the visual feature generation module is used to:

[0168] Performing feature extraction on the full information graph to obtain global features;

[0169] Performing feature extraction on the plurality of sub-graphs to obtain local features;

[0170] Inputting the global feature and the local feature into a visual encoder to obtain the visual feature;

[0171] The resolution of the full information graph, the resolution of the sub-graph and the input resolution of the visual encoder are the same.

[0172] In a possible implementation, the visual feature generation module is used to:

[0173] Arranging the local features and setting row labels based on the order of segmentation of the plurality of sub-graphs;

[0174] Flattening the global features and the local features after setting the row mark;

[0175] The global features after feature flattening are used as the query vector of cross attention, and the local features after feature flattening are used as the key vector and the value vector to obtain the visual features.

[0176] In a possible implementation, the output module is used to:

[0177] Mapping the visual features to input dimensions of the large language model;

[0178] The mapped visual features and the text features are input into the multimodal large model, and the model recognition results are output.

[0179] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0180] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0181] An embodiment of the present application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0182] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0183] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0185] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0186] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0187] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0190] Figure 8 This is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. Figure 8 As shown, the computer device of this embodiment includes: at least one processor 20 ( Figure 8 Only one is shown), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 implements the steps of any of the above-mentioned image processing method embodiments when executing the computer program 22.

[0191] The computer device may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 8 It is only an example of a computer device and does not constitute a limitation of the computer device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0192] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0193] In some embodiments, the memory 21 may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 21 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory 21 may also include both an internal storage unit of the computer device and an external storage device. The memory 21 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0194] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0195] The user personal information processed by the applicant will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, which may involve the user's account information, device information, driving information, vehicle information or other related information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0196] The Applicant attaches great importance to the security of user personal information and has adopted reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

Claims

1. An image processing method, characterized in that: include: Slicing the high-resolution image to obtain a plurality of sub-images, and resizing the high-resolution image to obtain a full information image of the high-resolution image; Based on the plurality of sub-graphs and the full information graph, obtaining visual features; obtaining text features associated with the high-resolution image; The visual features and the text features are input into a multimodal large model to obtain a model recognition result of the high-resolution image.

2. The image processing method according to claim 1, characterized in that: The step of segmenting the high-resolution image to obtain a plurality of sub-images, and resizing the high-resolution image to obtain a full information image of the high-resolution image includes: Determining a cutting strategy according to the length and width of the high-resolution image and the length and width of the predefined grid; Slicing the high-resolution image based on the slice strategy to obtain a plurality of sub-images; The high-resolution image is resized based on an input resolution of a visual encoder to generate a full information map; wherein the visual encoder is used to encode features of the full information map and the sub-map.

3. The image processing method according to claim 2, characterized in that: The determining of the image cutting strategy according to the length and width of the high-resolution image and the length and width of the predefined grid includes: Determine a first aspect ratio according to the length and width of the high-resolution image, and determine a second aspect ratio according to the length and width of the predefined grid; Obtaining a minimum value of a difference between the first aspect ratio and the second aspect ratio; If there is only one minimum value, determining the image cutting strategy based on the length and width of the predefined grid corresponding to the minimum value; If there are at least two minimum values, the image cutting strategy is determined according to the length and width of the high-resolution image, the length and width of the predefined grid corresponding to each minimum value, and the resolution of the sub-image.

4. The image processing method according to claim 2, wherein: The high-resolution image is segmented based on the segmentation strategy to obtain a plurality of sub-images, including: Adjusting the resolution of the high-resolution image to an integer multiple of the input resolution of the visual encoder; Based on the image cutting strategy, the high-resolution image after adjusting the resolution is cut into pieces to obtain a plurality of sub-images.

5. The image processing method according to any one of claims 1 to 4, characterized in that: The obtaining of visual features based on the plurality of sub-graphs and the full information graph includes: Performing feature extraction on the full information graph to obtain global features; Performing feature extraction on the plurality of sub-graphs to obtain local features; Inputting the global feature and the local feature into a visual encoder to obtain the visual feature; The resolution of the full information graph, the resolution of the sub-graph and the input resolution of the visual encoder are the same.

6. The image processing method according to claim 5, characterized in that: The step of inputting the global feature and the local feature into a visual encoder to obtain the visual feature comprises: Arranging the local features and setting row labels based on the order of segmentation of the plurality of sub-graphs; Flattening the global features and the local features after setting the row mark; The flattened global features are used as the query vector of the cross attention, and the flattened local features are used as the key vector and the value vector to obtain the visual features.

7. The image processing method according to any one of claims 1 to 4, characterized in that: The step of inputting the visual features and the text features into a multimodal large model to obtain a model recognition result of the high-resolution image includes: Mapping the visual features to input dimensions of a large language model; The mapped visual features and the text features are input into the multimodal large model, and the model recognition results are output.

8. An image processing device, characterized in that: include: A segmentation module is used to segment a high-resolution image into multiple sub-images; A size adjustment module, used for adjusting the size of the high-resolution image to obtain a full information map of the high-resolution image; A visual feature generation module, used for obtaining visual features based on the plurality of sub-graphs and the full information graph; A text feature acquisition module, used to acquire text features associated with the high-resolution image; The determination module is used to input the visual features and the text features into a multimodal large model to obtain a model recognition result of the high-resolution image.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer program product, characterized in that When the computer program product is executed on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 7.

Images