Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium, and is applied to various scenes such as artificial intelligence and large models, and the method comprises the steps: determining the texture complexity of a to-be-processed image; the resolution of the to-be-processed image is greater than a preset resolution threshold; determining a sliding window corresponding to the to-be-processed image according to the texture complexity; segmenting the to-be-processed image based on the sliding window to obtain a preset number of image areas corresponding to the to-be-processed image; inputting the preset number of image areas to a multi-modal large model for object recognition processing to obtain object recognition results corresponding to the preset number of image areas; and combining the object recognition results corresponding to the preset number of image areas to obtain an object recognition result corresponding to the to-be-processed image. According to the embodiment of the invention, the recognition precision of the object in the high-resolution to-be-processed image can be improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to an image processing method, apparatus, device and storage medium. Background Technology

[0002] Related technologies typically use traditional image processing algorithms, deep learning models, and fixed-size sliding windows to process high-resolution images.

[0003] However, traditional image processing algorithms, such as edge detection and feature extraction, are often limited by their complexity and cannot efficiently process large-size images. The detection accuracy of deep learning models such as Convolutional Neural Networks (CNNs) remains relatively low, especially when dealing with high-resolution images. Traditional deep learning models are limited by input size, making it impossible to directly process large images. Using a fixed-size sliding window to segment large-resolution images presents challenges. If the window size is too large, important image information may be cut out or lost; if the window size is too small, a large amount of window movement and overlapping calculations may be required, increasing computational complexity and time. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides an image processing method, apparatus, device, and storage medium.

[0005] On one hand, this application proposes an image processing method, the method comprising:

[0006] Determine the texture complexity of the image to be processed; the resolution of the image to be processed is greater than a preset resolution threshold;

[0007] The sliding window corresponding to the image to be processed is determined based on the texture complexity.

[0008] The image to be processed is segmented based on the sliding window to obtain a preset number of image regions corresponding to the image to be processed;

[0009] The preset number of image regions are input into a multimodal large model for object recognition processing to obtain object recognition results corresponding to each of the preset number of image regions; wherein, the multimodal large model is obtained by training a preset multimodal large model based on sample image regions and sample images, the sample image regions are obtained by segmenting the sample images based on sample sliding windows, and the sample sliding windows are determined based on the sample texture complexity of the sample images;

[0010] By combining the object recognition results corresponding to the preset number of image regions, the object recognition result corresponding to the image to be processed is obtained.

[0011] On the other hand, this application proposes an image processing apparatus, the apparatus comprising:

[0012] A texture complexity determination module is used to determine the texture complexity of the image to be processed; the resolution of the image to be processed is greater than a preset resolution threshold.

[0013] A sliding window determination module is used to determine the sliding window corresponding to the image to be processed based on the texture complexity.

[0014] The image region generation module is used to segment the image to be processed based on the sliding window to obtain a preset number of image regions corresponding to the image to be processed;

[0015] The object recognition result generation module is used to input the preset number of image regions into a multimodal large model for object recognition processing, and obtain the object recognition results corresponding to each of the preset number of image regions; wherein, the multimodal large model is obtained by training a preset multimodal large model based on sample image regions and sample images, the sample image regions are obtained by segmenting the sample images based on sample sliding windows, and the sample sliding windows are determined based on the sample texture complexity of the sample images;

[0016] The combination module is used to combine the object recognition results corresponding to the preset number of image regions to obtain the object recognition result corresponding to the image to be processed.

[0017] On the other hand, this application proposes an electronic device for image processing, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the image processing method as described above.

[0018] On the other hand, this application proposes a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the image processing method as described above.

[0019] On the other hand, this application proposes a computer program product, including a computer program that, when executed by a processor, implements the image processing method as described above.

[0020] This application proposes an image processing method, apparatus, device, and storage medium. The method involves: determining the texture complexity of the image to be processed; determining a sliding window corresponding to the image based on the texture complexity; segmenting the image based on the sliding window to obtain a preset number of image regions; inputting the preset number of image regions into a multimodal large model for object recognition processing to obtain object recognition results for each of the preset number of image regions; and combining the object recognition results for each of the preset number of image regions to obtain the object recognition result corresponding to the image to be processed. This achieves dynamic adjustment of the sliding window size based on the image's texture complexity and segmenting a high-resolution image into a preset number of image regions based on a determined sliding window size. This segmentation strategy can effectively process high-resolution images while preserving local image information, enabling the model to better learn and understand image details. Furthermore, using a trained multimodal large model for object recognition processing can improve the accuracy of object recognition processing, thereby improving the recognition accuracy of objects in high-resolution images without increasing hardware costs. Attached Figure Description

[0021] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram illustrating the implementation environment of an image processing method according to an exemplary embodiment.

[0023] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 .

[0024] Figure 3 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 2 .

[0025] Figure 4 This is a flowchart illustrating the training process of a multimodal large model according to an exemplary embodiment. Figure 1 .

[0026] Figure 5 This is a flowchart illustrating the training process of a multimodal large model according to an exemplary embodiment. Figure 2 .

[0027] Figure 6This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 3 .

[0028] Figure 7 A block diagram of an image processing apparatus is shown according to an exemplary embodiment.

[0029] Figure 8 This is a hardware structure block diagram of a server provided according to an exemplary embodiment. Detailed Implementation

[0030] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0031] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0032] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0033] Optionally, the process of segmenting the image to be processed and recognizing objects in the image region in the embodiments of this application involves technologies such as image processing, image recognition, and image semantic understanding in computer vision.

[0034] Pre-trained models, also known as foundational models or large models, refer to deep neural networks (DNNs) with a large number of parameters. These DNNs are trained on massive amounts of unlabeled data. Leveraging the function approximation capabilities of large-parameter DNNs, Proximity-Based Transformers (PTMs) extract common features from the data. Through fine-tuning, parameter efficient fine-tuning (PEFT), and prompt-tuning techniques, they are suitable for downstream tasks. Therefore, pre-trained models can achieve ideal results in small-shot or zero-shot scenarios. PTMs can be categorized according to the data modality they process, such as language models (ELMO, BERT, GPT), visual models (Swin-transformer, ViT, V-MOE), speech models (VALL-E), and multimodal models (ViBERT, CLIP, Flamingo, Gato). Multimodal models refer to models that establish feature representations for two or more data modalities. Pre-trained models are important tools for outputting AI-generated content (AIGC) and can also serve as a general interface connecting multiple task-specific models.

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0037] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] Figure 1 This is a schematic diagram illustrating an implementation environment for an image processing method according to an exemplary embodiment. For example... Figure 1 As shown, the implementation environment may include at least terminal 01 and server 02. The terminal 01 and server 02 may be directly or indirectly connected through wired or wireless communication. This embodiment of the application does not impose any limitations on this.

[0039] Specifically, server 02 can be used to process the image to be processed and obtain object recognition results. Optionally, server 02 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0040] Specifically, the terminal 01 can be used to display the object recognition results. The terminal 01 may include, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.

[0041] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0042] It should be noted that, Figure 1 This is just one example. Other implementation environments may also be included in other scenarios.

[0043] It should be noted that in the specific implementation of this application, user information, such as images to be processed and other related data, is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0044] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 This method can be used for Figure 1 In the implementation environment described in this specification, the method operation steps are as illustrated in the embodiments or flowcharts. However, based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the embodiments or drawings... Figure 2 As shown, the method may include:

[0045] S101. Determine the texture complexity of the image to be processed; the resolution of the image to be processed is greater than a preset resolution threshold.

[0046] Optionally, the image to be processed is an image with a resolution greater than a preset resolution threshold, i.e., a high-resolution image. Here, high resolution refers to a larger size, i.e., a larger length and width. It should be noted that the embodiments of this application do not limit the specific type of the image to be processed; it can correspond to actual business scenarios. For example, in a component identification scenario for drawings, the image to be processed can be the drawing itself. As another example, in a small target detection scenario, the image to be processed can be various images including small targets.

[0047] Optionally, the texture complexity is used to characterize the complexity of the texture features of the image to be processed. For example, the texture complexity can be represented by the variance of the texture features of the image to be processed. The larger the variance, the more complex the texture and the higher the texture complexity; the smaller the variance, the less complex the texture and the lower the complexity.

[0048] S103. Determine the sliding window corresponding to the image to be processed based on the texture complexity.

[0049] Optionally, a mapping relationship between texture complexity and sliding window can be pre-established, and the corresponding sliding window can be dynamically determined for each image to be processed based on this mapping relationship. For example, the texture complexity and the size of the sliding window can be inversely proportional; the greater the texture complexity, the smaller the sliding window can be selected, and the smaller the texture complexity, the larger the sliding window can be selected.

[0050] It should be noted that, since the texture complexity of different images to be processed is different, the size of the sliding window for different images to be processed may be different. That is, the size of the sliding window corresponding to different images to be processed is not fixed. This realizes the dynamic adjustment of the size of the sliding window of the image according to the texture complexity of the image, so as to divide the high-resolution image to be processed into image regions of fixed size, effectively solving the problem of large image resolution.

[0051] S105. Segment the image to be processed based on a sliding window to obtain a preset number of image regions corresponding to the image to be processed.

[0052] In this embodiment, after determining that a sliding window has been obtained, the sliding window can be used to slide on the image to be processed in order to segment the image to be processed and obtain a preset number of image regions of fixed size corresponding to the image to be processed.

[0053] It should be noted that the embodiments of this application do not limit the sliding direction or sliding method of the sliding window.

[0054] Since the sliding window is dynamically adjusted based on the texture complexity of the image to be processed, the high-resolution image is divided into a preset number of image regions according to the sliding window of a certain size. This segmentation strategy can effectively process high-resolution images while preserving the local information of the image, enabling the model to better learn and understand the details of the image.

[0055] S107. Input a preset number of image regions to a multimodal large model for object recognition processing, and obtain the object recognition results corresponding to each preset number of image regions; wherein, the multimodal large model is obtained by training a preset multimodal large model based on sample image regions and sample images, the sample image regions are obtained by segmenting the sample images based on sample sliding windows, and the sample sliding windows are determined based on the sample texture complexity of the sample images.

[0056] In this embodiment, a multimodal large model can be pre-trained based on sample image regions and sample images to obtain the multimodal large model. The sample image regions are obtained by sliding a sample sliding window across the sample image to segment it, and the sample sliding window is determined based on the sample texture complexity of the sample image.

[0057] Optionally, the sample image has a resolution greater than a preset resolution threshold, meaning the sample image is a high-resolution image. Alternatively, the sample image includes images with a resolution greater than the preset resolution threshold.

[0058] It should be noted that the method for determining the texture complexity of this sample is the same as the method for determining the texture complexity described above, the method for determining the sliding window of this sample is the same as the method for determining the sliding window described above, and the method for determining the image region of this sample is the same as the method for determining the image region described above. No specific limitations are made here.

[0059] After segmenting the image to be processed to obtain a preset number of image regions, the preset number of image regions can be input into a multimodal large model for object recognition processing to obtain the object recognition results corresponding to each of the preset number of image regions.

[0060] It should be noted that the type of object recognition result can be determined based on the actual business scenario, and there are no specific limitations on it. For example, in a scenario of component recognition from a drawing, the image to be processed can be the drawing itself, and the object recognition result can include the category of the component in the drawing, as well as the component's location information (i.e., coordinate information) within the drawing. As another example, in a target detection scenario, the image to be processed can be various images containing the target to be processed, and the object recognition result can be the target's category, as well as the target's location information within the image to be processed.

[0061] Optionally, in the embodiments of this application, "multimodal model" refers to modality transformation, that is, the behavior of mapping one modality to another. Accordingly, the "multimodal model" refers to a modality transformation model, that is, a model that maps one modality to another.

[0062] In one implementation, for a component recognition scenario on a drawing, the object recognition result may include the category of the component in the drawing and the component's location information (i.e., coordinate information) in the drawing. Since the component's category and location information can be represented by text, "multimodal" can refer to the behavior of mapping the input image modality to a text modality. Correspondingly, the "multimodal model" refers to a model that maps the input image modality to a text modality.

[0063] S109. Combine the object recognition results corresponding to each of the preset number of image regions to obtain the object recognition result corresponding to the image to be processed.

[0064] In this embodiment, after obtaining the object recognition results corresponding to each of the preset number of image regions, the object recognition results corresponding to each of the preset number of image regions can be combined to obtain the object recognition result corresponding to the image to be processed.

[0065] Thus, the size of the sliding window of an image is dynamically adjusted according to the texture complexity of the image, and the high-resolution image is divided into a preset number of image regions according to the sliding window of a determined size. This segmentation strategy can effectively process high-resolution images while preserving the local information of the image, enabling the model to better learn and understand the details of the image. In addition, using the trained multimodal large model to perform object recognition processing on the image can improve the accuracy of object recognition processing, thereby improving the recognition accuracy of the multimodal large model for high-resolution images without increasing hardware costs.

[0066] It should be noted that in step S101 above, the determination of the texture complexity of the image to be processed can be achieved in a variety of ways, and no specific limitation is made.

[0067] In one implementation, a texture complexity processing model can be pre-trained, and the texture complexity of the image to be processed can be obtained by processing the texture complexity of the image to be processed.

[0068] In another implementation, Figure 3 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 2 ,like Figure 3 As shown, in step S101 above, determining the texture complexity of the image to be processed may include:

[0069] S1011. Determine the preset direction, preset frequency, and standard deviation.

[0070] S1013. Combine the preset direction and preset frequency to obtain a preset number of Gaussian functions.

[0071] S1015. Convolve the image to be processed with each Gaussian function to obtain the response image corresponding to each Gaussian function.

[0072] S1017. Perform texture feature statistical processing on the response image corresponding to each Gaussian function to generate texture features corresponding to each Gaussian function.

[0073] S1019. Generate the texture complexity of the image to be processed based on the variance of the texture features corresponding to each Gaussian function.

[0074] In this embodiment, the texture complexity of the image to be processed can be determined using a Gabor filter. A Gabor filter is a two-dimensional Gaussian function modulated by a sine function, expressed as follows: G(x,y)=exp(-((x^2+y^2) / (2*σ^2)))*exp(j*(2π*(u*x+v*y))), where: x and y are the coordinates in the image to be processed; σ is the standard deviation of the Gaussian function, which determines the spatial bandwidth of the filter; u and v are the horizontal and vertical frequencies of the filter, respectively; and j represents the imaginary unit.

[0075] In practical applications, a preset number of Gabor filters with different directions and frequencies can be used to capture texture information in different directions in the image to be processed.

[0076] When using a Gabor filter to calculate the texture complexity of an image, the main parameter settings and the calculation process are as follows:

[0077] I. Parameter Settings:

[0078] 1. Select a preset direction. For example, you can select four preset directions: 0°, 45°, 90°, and 135°.

[0079] 2. Determine the preset frequency. For example, you can choose two preset frequencies: 0.1 cycles / pixel and 0.5 cycles / pixel. Here, cycles / pixel refers to the period per pixel, 0.1 cycles / pixel refers to the horizontal frequency, and 0.5 cycles / pixel refers to the vertical frequency.

[0080] 3. Determine the standard deviation. For example, you can set the standard deviation σ = 10 pixels, which is the standard deviation of the pixel range covered by the filter.

[0081] II. Generating Gabor Filter Banks

[0082] After setting the above parameters, for each preset direction, it can be combined with a preset frequency to obtain a preset number of Gaussian functions, i.e., a preset number of Gabor filters. For example, combining 0° with the above two preset frequencies according to the above expression yields one Gabor filter; combining 45° with the above two preset frequencies according to the above expression yields one Gabor filter; combining 90° with the above two preset frequencies according to the above expression yields one Gabor filter; and combining 135° with the above two preset frequencies according to the above expression yields one Gabor filter.

[0083] III. Image Preprocessing: Convert the image to grayscale and remove noise from the image.

[0084] IV. Calculating the Gabor Filter Response

[0085] The image to be processed is convolved with each Gaussian function to obtain the response image corresponding to each Gaussian function. This response image can be considered as a texture feature image. Here, the Gabor filter can be considered as a filtering matrix, and convolution is equivalent to multiplying the image and the filtering matrix element by element and then summing the results.

[0086] V. Extracting Texture Features

[0087] For the response image corresponding to each Gaussian function, texture feature statistics can be performed to generate the texture features corresponding to each Gaussian function.

[0088] VI. Calculate texture complexity

[0089] The variance of the texture features corresponding to each Gaussian function can be calculated, and the texture complexity of the image to be processed can be generated based on this variance.

[0090] Therefore, by using a preset number of Gaussian functions with different directions and frequencies, texture information in different directions in the image to be processed can be captured, which improves the accuracy of texture feature extraction and texture complexity calculation, thereby improving the determination accuracy of the sliding window and thus improving the recognition accuracy of the object recognition result.

[0091] In an optional embodiment, in step S1017 above, the above-mentioned textural feature statistical processing of the response image corresponding to each Gaussian function to generate the texture feature corresponding to each Gaussian function may include:

[0092] The mean and standard deviation of the response image corresponding to each Gaussian function are determined to obtain the texture features corresponding to each Gaussian function.

[0093] In this embodiment, since the response image can be considered as a texture feature image, for each response image corresponding to a Gaussian function, its mean μ and standard deviation σ can be calculated, and the mean μ and standard deviation σ can be used as the final representation of the texture feature.

[0094] In other implementations, a pre-trained texture feature extraction model can be used to extract texture features from the response function corresponding to each Gaussian function, thereby obtaining the texture features corresponding to each Gaussian function.

[0095] Accordingly, in step S1019 above, generating the texture complexity of the image to be processed based on the variance of the texture features corresponding to each Gaussian function may include:

[0096] The mean and standard deviation of the response image corresponding to each Gaussian function are combined to obtain the characteristic vector.

[0097] The variance of the eigenvectors is obtained by calculating the variance of the response image corresponding to each Gaussian function, including the mean and standard deviation, the mean of the eigenvectors, and the number of Gaussian functions.

[0098] The texture complexity of the image to be processed is evaluated based on the variance of the feature vectors.

[0099] In this embodiment, the mean and standard deviation of the response image corresponding to each Gaussian function can be combined into a feature vector, for example, F = [μ1,σ1,μ2,σ2,...,μn,σn], where n is the number of Gabor filters, i.e. the number of Gaussian functions, and μn and σn refer to the mean and standard deviation of the response image corresponding to the nth Gaussian function.

[0100] Next, the variance of the eigenvectors is calculated by examining the mean and standard deviation of the response image corresponding to each Gaussian function, the mean of the eigenvectors, and the number of Gaussian functions. For example, the formula for calculating the variance can be as follows:

[0101] Var(F)=(1 / (2n))*Σ(Fi-μF)^2;

[0102] Where Fi is the i-th element (i.e., μi, σi) of the feature vector F, where μi and σi refer to the mean and standard deviation of the response image corresponding to the i-th Gaussian function, respectively; μF is the mean of F; and n is the number of Gabor filters, i.e., the number of Gaussian functions.

[0103] Finally, the texture complexity of the image to be processed is generated based on the variance of the feature vectors. For example, the variance of the feature vectors is directly proportional to the texture complexity; the larger the variance of the feature vectors, the higher the texture complexity of the image to be processed, and the smaller the variance of the feature vectors, the lower the texture complexity of the image to be processed.

[0104] Since the variance of the feature vector is calculated by taking the variance of the mean and standard deviation of the response image corresponding to each Gaussian function, the mean of the feature vector, and the number of Gaussian functions, and since the mean and standard deviation of the response image corresponding to each Gaussian function can characterize the texture feature complexity of the image to be processed, the variance of the feature vector calculated in the above way can also fully characterize the texture feature complexity of the image to be processed. Therefore, by using the variance of this feature vector to evaluate the texture complexity of the image to be processed, the evaluation accuracy of the texture complexity of the image to be processed can be improved, thereby improving the determination accuracy of the sliding window, and thus improving the recognition accuracy of the object recognition result.

[0105] In an optional embodiment, continue as follows Figure 3 As shown, in step S103 above, the sliding window corresponding to the image to be processed based on texture complexity may include:

[0106] S1031. Obtain the preset variance threshold; the preset variance threshold is determined based on the distribution of the variance of the texture complexity of the sample image;

[0107] S1033. When the texture complexity is greater than or equal to a preset variance threshold, the sliding window corresponding to the image to be processed is determined as the first sliding window, and when the texture complexity is less than the preset variance threshold, the sliding window corresponding to the image to be processed is determined as the second sliding window; wherein, the size of the first sliding window is smaller than the size of the second sliding window.

[0108] In this embodiment, a preset variance threshold can be determined in advance based on the distribution of the variance of the texture complexity of the sample image. It should be noted that the process of determining the texture complexity of the sample image is similar to the process of determining the texture complexity of the image to be processed, i.e., it involves parameter setting, generating a Gabor filter bank, calculating the Gabor filter response, extracting texture features, and calculating the texture complexity. Here, "variance of the texture complexity of the sample image" refers to the variance of the feature vector F during the process of calculating texture complexity.

[0109] Assume there are M sample images, where M is a positive integer greater than 1. The variance of the feature vector F corresponding to each sample image can be calculated as described above, resulting in M ​​variances. These M variances are then arranged in descending or ascending order, and the variance at the middle position of this arrangement is taken as the preset variance threshold.

[0110] For the image to be processed, its texture complexity can be compared with a preset variance threshold. If the texture complexity is greater than or equal to the preset variance threshold, it indicates that the image contains more details, and a smaller first sliding window can be used, for example, a 786x786 first sliding window. If the texture complexity is less than the preset variance threshold, it indicates that the image is relatively smooth, and a larger second sliding window can be used, for example, a 1536x1536 second sliding window.

[0111] Since the distribution of the variance of texture complexity in the sample image can fully reflect the distribution of texture complexity variance, the preset variance threshold determined based on this variance distribution can accurately characterize the texture complexity of the sample image. Therefore, by comparing the preset variance threshold with the texture complexity of the image to be processed to determine the sliding window, the determined sliding window can fully consider the texture complexity of the image to be processed. This achieves dynamic adjustment of the size of the sliding window of the image based on the texture complexity of the image, so as to divide the high-resolution image to be processed into image regions of fixed size, effectively solving the problem of large image resolution. Moreover, dynamically adjusting the size of the sliding window of the image based on the texture complexity of the image can preserve the local information of the image during the segmentation process, further improving the recognition accuracy of the object recognition result.

[0112] In an optional embodiment, in step S105 above, the segmentation of the image to be processed based on a sliding window to obtain a preset number of image regions corresponding to the image to be processed may include:

[0113] The image to be processed is segmented by sliding a sliding window to obtain a preset number of image regions.

[0114] In two adjacent sliding processes, the sliding windows partially overlap.

[0115] In this embodiment, to avoid information entropy loss in the segmented regions and reduce boundary effects, the sliding windows in adjacent sliding processes can partially overlap. For example, a fixed proportion of the overlapping area can be set. It should be noted that this fixed proportion can be set according to actual business needs and is not specifically limited thereto.

[0116] Optionally, during the first sliding process, a starting point can be set on the image to be processed, and the sliding window can be placed at the starting point to perform the first segmentation of the image to be processed. Then, the sliding window is moved in a preset direction and stops when it reaches a preset position. At this time, the sliding window overlaps with the first sliding window, and the image to be processed is segmented a second time through the sliding window. Then, the sliding window is moved in a preset direction and stops when it reaches a preset position. At this time, the sliding window overlaps with the second sliding window, and the image to be processed is segmented a third time through the sliding window, and so on, until the image to be processed is completely segmented to obtain a preset number of image regions.

[0117] In an optional embodiment, the aforementioned multimodal large model includes a visual encoder and a text decoder. Therefore, in step S107, inputting a preset number of image regions into the multimodal large model for object recognition processing to obtain object recognition results corresponding to each of the preset number of image regions may include:

[0118] Input a preset number of image regions to the visual encoder for visual feature transformation, and obtain the visual feature vectors corresponding to each preset number of image regions.

[0119] Input the visual feature vectors corresponding to a preset number of image regions to the text decoder to generate token sequences, thus obtaining the token sequences corresponding to the preset number of image regions.

[0120] The token sequences corresponding to a preset number of image regions are converted into a new format to obtain the object recognition results corresponding to the preset number of image regions.

[0121] In this embodiment, the multimodal large model may include a visual encoder and a text decoder.

[0122] The visual encoder transforms the input image region x∈RH×W×C into a set of embedding vectors {zi|zi∈Rd,1≤i≤n}, where n is the number of image regions and d is the dimension of the encoder's latent vectors. These embedding vectors can be considered visual feature vectors. The output {z} from the encoder module is then fed into the decoder.

[0123] The text decoder, for a given {z} from the visual encoder, generates a token sequence (yi)mi = 1, where yi ∈ Rv is the one-hot vector of the i-th token, v is the size of the token vocabulary, and m is a hyperparameter. The output token sequence is then format-converted to obtain the object recognition results for a preset number of image regions. Taking component recognition as an example, the object category and location information of the component can be obtained.

[0124] As can be seen, the visual encoder converts the input image modality into a visual feature vector, while the text decoder converts the visual feature vector into a text modality, thereby realizing the mapping of the input image modality to the text modality through a multimodal model.

[0125] Since the multimodal large model includes a visual encoder and a text decoder, the visual encoder can convert the input image region into a set of visual embedding vectors, while the text decoder can convert the visual embedding vectors into a token sequence and obtain the object recognition result based on the token sequence, thereby improving the recognition accuracy of the object recognition result.

[0126] In an optional embodiment, the object recognition results corresponding to each of the preset number of image regions include the object location information corresponding to each of the preset number of image regions. After performing object recognition processing on the preset number of image regions using a multimodal large model to obtain the object recognition results corresponding to each of the preset number of image regions, the method may further include:

[0127] Obtain the original position information of the top left corner of each image region in the image to be processed.

[0128] Based on the object location information and original location information corresponding to each of the preset number of image regions, the target location information of each of the preset number of image regions in the image to be processed is generated.

[0129] Optionally, in order to restore the object recognition result to the image to be processed and thus improve the recognition accuracy of the object recognition result, during the process of segmenting the image to be processed, the original position information (i.e., coordinate information) of the upper left corner of each image region in the image to be processed can be stored, and the position information can be used as an offset, which is used to restore the object position information to the image to be processed.

[0130] Accordingly, after obtaining the object position information corresponding to each of the preset number of image regions, the object position information can be added with the corresponding offset to obtain the target position information of each of the preset number of image regions in the image to be processed, so as to restore the original position information identified to the image to be processed.

[0131] Taking the object location information as a coordinate frame as an example, a corresponding offset can be added to the coordinate frame position to restore the object recognition result to the image to be processed.

[0132] In an optional embodiment, the object recognition result includes object category and object location information. Therefore, in step S109, combining the object recognition results corresponding to a preset number of image regions to obtain the object recognition result corresponding to the image to be processed may include:

[0133] The object categories corresponding to a preset number of image regions are combined to obtain the object category to which the object in the image to be processed belongs. The object location information corresponding to a preset number of image regions is combined to obtain the object location information of the object in the image to be processed.

[0134] Based on the object category and object location information of the objects in the image to be processed, the object recognition result corresponding to the image to be processed is generated.

[0135] In this embodiment, when the object recognition result includes object category and object location information, for the object category, the object categories corresponding to each of a preset number of image regions can be combined to obtain the object category to which the object in the image to be processed belongs. For the object location information, the object location information corresponding to each of a preset number of image regions can be combined to obtain the object location information of the object in the image to be processed.

[0136] Taking the application scenario as the component in the inspection drawing, the object category as the component category, and the object location information as the location information of the component in the inspection drawing, when combining, the component categories corresponding to each of the preset number of image regions can be combined to obtain the component category to which the component in the image to be processed belongs, and the component location information corresponding to each of the preset number of image regions can be combined to obtain the component location information of the component in the image to be processed.

[0137] Since the object category corresponding to each image region is obtained by recognizing each image region based on a multimodal large model, the recognition accuracy of the object category corresponding to each image region is relatively high. Furthermore, each image region is segmented based on a sliding window with dynamically determined texture complexity. Segmenting the image based on a sliding window with dynamically determined texture complexity can preserve the local information of the image, enabling the model to better learn and understand the details of the image, thereby further improving the recognition accuracy of the object category corresponding to each image region. On this basis, combining the object category and object location information corresponding to each image region can improve the recognition accuracy of the object recognition result corresponding to the image to be processed.

[0138] In an optional embodiment, in step S109 above, combining the object recognition results corresponding to each of the preset number of image regions to obtain the object recognition result corresponding to the image to be processed may include:

[0139] The overlapping object recognition result is determined from all object recognition results corresponding to a preset number of image regions.

[0140] Remove overlapping object recognition results from all object recognition results corresponding to a preset number of image regions to obtain candidate object recognition results.

[0141] The candidate object recognition results are combined to obtain the object recognition result corresponding to the image to be processed.

[0142] In this embodiment, in order to effectively remove overlapping object recognition results and further improve the accuracy of object recognition results, overlapping object recognition results can be determined from all object recognition results corresponding to a preset number of image regions, and the overlapping object recognition results can be deleted to obtain candidate object recognition results. Finally, the candidate object recognition results are combined to obtain the final object recognition result.

[0143] It should be noted that when the object recognition result includes object category and object location information, overlapping object recognition result means that both object category and object location information overlap.

[0144] In one implementation, non-maximum suppression (NMS) can be used to remove the overlapping object identification results. Specifically, NMS may include:

[0145] When the object recognition result includes object category and object location information, the object location information can be a coordinate bounding box.

[0146] First, sort all bounding boxes according to their confidence scores, with the box having the highest confidence score at the top. Starting with the box with the highest confidence score, iterate through the remaining bounding boxes in turn. For the currently encountered bounding box, if its overlap with previously retained bounding boxes (calculated by the Intersection over Union) exceeds a certain threshold, it is suppressed and not retained. Continue iterating through the next bounding box, repeating the above process until all bounding boxes have been processed.

[0147] Therefore, NMS can suppress a large number of overlapping bounding boxes, retaining only the best bounding box, thus obtaining the final candidate object recognition result.

[0148] The following describes the training process of a multimodal large model:

[0149] Figure 4 This is a flowchart illustrating the training process of a multimodal large model according to an exemplary embodiment. Figure 1 ,like Figure 4 As shown, the training process may include:

[0150] S201. Determine the sample texture complexity of the sample image.

[0151] Optionally, the sample image is an image with a resolution greater than a preset resolution threshold, i.e., a high-resolution image. High resolution refers to a larger size, i.e., a larger length and width. It should be noted that the embodiments of this application do not limit the specific type of the sample image; it can correspond to actual business scenarios. For example, in a component recognition scenario using drawings, the sample image can be the drawing itself. As another example, in a small target detection scenario, the sample image can be various images including small targets.

[0152] Optionally, the sample texture complexity is used to characterize the complexity of the texture features of the sample image. For example, the sample texture complexity can be represented by the variance of the texture features of the sample image. The larger the variance, the more complex the sample texture and the higher the texture complexity; the smaller the variance, the less complex the sample texture and the lower the complexity.

[0153] S203. Determine the sample sliding window corresponding to the sample image based on the sample texture complexity.

[0154] Optionally, a mapping relationship between sample texture complexity and sample sliding window can be pre-established. Based on this mapping relationship, the corresponding sample sliding window is dynamically determined for each sample image. For example, the sample texture complexity and the size of the sample sliding window can be inversely proportional; the greater the sample texture complexity, the smaller the sample sliding window can be selected, and the smaller the sample texture complexity, the larger the sample sliding window can be selected.

[0155] It should be noted that, due to the different texture complexities of different sample images, the size of the sample sliding window for different sample images may vary. That is, the size of the sample sliding window for different sample images is not fixed. This enables the dynamic adjustment of the size of the sample sliding window based on the texture complexity of the sample image, so as to divide the high-resolution sample image into image regions of fixed size, effectively solving the problem of large image resolution.

[0156] S205. Segment the sample image based on the sample sliding window to obtain a preset number of sample image regions corresponding to the sample image.

[0157] Figure 5 This is a flowchart illustrating the training process of a multimodal large model according to an exemplary embodiment. Figure 2 ,like Figure 5 As shown, after obtaining the sample sliding window, this window can be used to slide across the sample image to segment the image and obtain a preset number of sample image regions. Figure 5 P1, P2, ..., Pn are shown in the diagram.

[0158] It should be noted that the embodiments of this application do not limit the sliding direction or sliding method of the sample sliding window.

[0159] Since the sample sliding window is dynamically adjusted based on the sample texture complexity of the sample image, using the sample sliding window to segment the sample image can avoid information loss in the segmented region, reduce boundary effects, and avoid a large number of window movements and overlapping calculations, thus reducing computational complexity and time.

[0160] S207. Input a preset number of sample image regions, sample images, and preset text prompts into a preset multimodal large model for object recognition processing, and obtain the predicted object recognition results corresponding to each of the preset number of sample image regions.

[0161] In this embodiment, the segmented sample image region and the sample image can be scaled together to a fixed size and used as input to a preset multimodal large model. At the same time, in order to improve the model prediction accuracy, preset text prompts can also be input.

[0162] Optionally, similar to the "multimodal model" mentioned above, the "multimodal" in the "preset multimodal model" in this application embodiment also refers to mode transformation, that is, the behavior of mapping one mode to another mode. Accordingly, the "preset multimodal model" also refers to the mode transformation model, that is, the model of mapping one mode to another mode.

[0163] In one implementation, for a component recognition scenario in a drawing, the predicted object recognition result may include the category of the component in the drawing and the component's location information (i.e., coordinate information) in the drawing. Since the component's category and location information can be represented by text, "multimodal" also refers to the behavior of mapping the input image modality to the text modality. Correspondingly, the "preset multimodal model" also refers to the model that maps the input image modality to the text modality.

[0164] Continue as Figure 5 As shown, this pre-defined multimodal large model includes a visual encoder and a text decoder. The visual encoder converts the input sample image region x∈RH region e into a set of embedding vectors {zi|zi∈Rd,1}, where n is the number of sample image regions and d is the dimension of the encoder's latent vectors. These embedding vectors can be considered as visual feature vectors. The output {z} from the encoder module is then input into the text decoder.

[0165] The text decoder, given {z} by the visual encoder, generates a token sequence (yi)mi = 1, where yi ∈ Rv is the one-hot vector of the i-th token, v is the size of the token vocabulary, and m is a hyperparameter. The output token sequence is then converted into the desired structured format to obtain the object recognition result.

[0166] The preset text prompt: A prompt will be input into the text decoder. Taking a component recognition scenario as an example, the preset text prompt could be: <question>Current transformer and its location <answer>Given a prompt, a token sequence can be generated. This token sequence is input into the decoder and aggregated with visual features (i.e., features output by the visual encoder) to obtain the predicted object recognition results for each sample image region. Taking a component recognition scenario as an example, the predicted object recognition scenario can include component category and component location information. S209. Determine the loss data based on the predicted object recognition results and the preset text prompt information.

[0167] Since the preset text prompt information can be: <question>Current transformer and its location <answer>Therefore, the preset text prompt can be considered as a pre-given object recognition result. After the text decoder outputs the predicted object recognition result, the predicted object recognition result predicted by the model can be compared with the preset text prompt to obtain the difference between the predicted object recognition result and the preset text prompt. The loss data is then calculated based on this difference.

[0168] Optionally, if the object recognition result includes object category and object location information, the object category can be compared with the object category indicated in the preset text prompt to obtain the object category difference, the object location information can be compared with the object location information indicated in the preset text prompt to obtain the object location information difference, and the loss data can be determined based on the object category difference and the object location information difference.

[0169] For example, the above difference can be calculated using at least one of KL divergence, cross-entropy, or squared difference loss. KL divergence is a statistic that measures the difference between two probability distributions.

[0170] S2011. Adjust the model parameters of the preset multimodal large model according to the loss data until the preset multimodal large model converges, and obtain the trained multimodal large model.

[0171] Optionally, after calculating the loss data, the model parameters of the preset multimodal large model can be continuously adjusted based on the high loss data. That is, the model parameters of the visual encoder and the text decoder can be continuously adjusted until the loss data between the predicted object recognition result output by the preset multimodal large model and the preset text prompt information is less than a preset threshold, or the training times of the preset multimodal large model reach a preset threshold, thus obtaining the trained multimodal large model.

[0172] This application embodiment uses a pre-defined multimodal large model for object recognition, which includes an encoder and a decoder structure. First, during the model training phase, high-resolution sample images are segmented into fixed-size sample image regions according to a sample sliding window of a defined size. This segmentation strategy effectively processes high-resolution sample images while preserving local image information, enabling the pre-defined multimodal large model to better learn and understand image details. These segmented sample image regions, along with the original sample images, are scaled to a fixed size for fine-tuning the training of the pre-defined multimodal large model. In this way, the pre-defined multimodal large model can learn and understand images from different perspectives, thereby improving model performance and generalization ability, and ultimately enhancing the recognition accuracy of object recognition results.

[0173] It should be noted that in step S201 above, the method of determining the sample texture complexity of the sample image is the same as that of determining the texture complexity of the image to be processed in step S101 above. For details, please refer to steps S1011-S1019 above, which will not be repeated here.

[0174] It should be noted that in step S203 above, the process of determining the sample sliding window corresponding to the sample image based on the sample texture complexity is the same as the process of determining the sliding window corresponding to the image to be processed in step S103 above. For details, please refer to steps S1031-S1033 above, which will not be repeated here.

[0175] It should be noted that in step S205 above, the segmentation of the sample image based on the sample sliding window to obtain a preset number of sample image regions corresponding to the sample image is the same as the determination of a preset number of image regions corresponding to the image to be processed in step S105 above. For details, please refer to step S105 above, which will not be repeated here.

[0176] The following example, using a component recognition scenario based on drawings as an example, where the object recognition result includes component category and component location information, illustrates the above image processing method:

[0177] 1) Train a large multimodal model. Please refer to steps S201-S2011 above for the training process, which will not be repeated here.

[0178] 2) Obtain the drawing to be inspected, whose resolution is greater than the preset resolution threshold.

[0179] 3) Determine the texture complexity of the drawing to be inspected. Please refer to steps S1011-S1019 above for this process, which is not specifically limited.

[0180] 4) Determine the sliding window corresponding to the drawing to be inspected based on the texture complexity. Please refer to steps S1031-S1033 above for this process, which is not specifically limited.

[0181] 5) Figure 6 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 3 ,like Figure 6 As shown, the image to be processed can be segmented based on a sliding window to obtain a preset number of image regions corresponding to the image to be processed, such as... Figure 6 The coordinates P1, P2, ..., Pn are shown in the diagram. Please refer to step S105 above for this process; no specific limitations are imposed. Simultaneously, to restore the components to the drawing to be inspected, the coordinates of the upper left corner of each image area in the drawing to be inspected can be stored as an offset.

[0182] 6) Continue as follows Figure 6 As shown, a preset number of image regions are input into a multimodal large model for object recognition processing, resulting in object recognition results for each preset number of image regions (i.e., component categories and component location information corresponding to each preset number of image regions). This process is described in step S107 above, and is not specifically limited to this step.

[0183] 7) Add the corresponding offset to the component position information corresponding to each of the preset number of image regions to obtain the final component position information.

[0184] 8) Continue as follows Figure 6 As shown, non-maximum suppression (NMS) is used to remove duplicate components identified at overlapping locations, resulting in the final object identification result (i.e., the final component category and component location information).

[0185] Figure 7 A block diagram of an image processing apparatus is shown according to an exemplary embodiment, such as Figure 7 The image processing apparatus includes:

[0186] Texture complexity determination module 301 is used to determine the texture complexity of the image to be processed; the resolution of the image to be processed is greater than a preset resolution threshold.

[0187] The sliding window determination module 303 is used to determine the sliding window corresponding to the image to be processed based on the texture complexity.

[0188] The image region generation module 305 is used to segment the image to be processed based on the sliding window to obtain a preset number of image regions corresponding to the image to be processed.

[0189] The object recognition result generation module 307 is used to input the preset number of image regions into a multimodal large model for object recognition processing, and obtain the object recognition results corresponding to each of the preset number of image regions; wherein, the multimodal large model is obtained by training a preset multimodal large model based on sample image regions and sample images, the sample image regions are obtained by segmenting the sample images based on sample sliding windows, and the sample sliding windows are determined based on the sample texture complexity of the sample images;

[0190] The combination module 309 is used to combine the object recognition results corresponding to the preset number of image regions to obtain the object recognition result corresponding to the image to be processed.

[0191] In an optional embodiment, the sliding window determination module includes:

[0192] A preset variance threshold acquisition unit is used to acquire a preset variance threshold; the preset variance threshold is determined based on the distribution of the variance of the texture complexity of the sample image.

[0193] A sliding window generation unit is configured to determine a first sliding window corresponding to the image to be processed when the texture complexity is greater than or equal to the preset variance threshold, and to determine a second sliding window corresponding to the image to be processed when the texture complexity is less than the preset variance threshold; wherein the size of the first sliding window is smaller than the size of the second sliding window.

[0194] In an optional embodiment, the texture complexity determination module includes:

[0195] The parameter determination unit is used to determine the preset direction, preset frequency, and standard deviation;

[0196] The direction-frequency combination unit is used to combine the preset direction and preset frequency to obtain a preset number of Gaussian functions;

[0197] A convolutional unit is used to convolve the image to be processed with each Gaussian function to obtain the response image corresponding to each Gaussian function;

[0198] The statistical unit is used to perform texture feature statistical processing on the response image corresponding to each Gaussian function to generate texture features corresponding to each Gaussian function.

[0199] The variance processing unit is used to generate the texture complexity of the image to be processed based on the variance of the texture features corresponding to each Gaussian function.

[0200] In an optional embodiment, the statistical unit includes:

[0201] The texture feature generation subunit is used to determine the mean and standard deviation of the response image corresponding to each Gaussian function, and to obtain the texture features corresponding to each Gaussian function;

[0202] Accordingly, the variance processing unit includes:

[0203] The feature vector generation subunit is used to combine the mean and standard deviation of the response image corresponding to each Gaussian function to obtain the feature vector;

[0204] The contrast calculation subunit is used to calculate the variance of the response image corresponding to each Gaussian function, the mean and standard deviation of the feature vector, and the number of Gaussian functions to obtain the variance of the feature vector.

[0205] An evaluation subunit is used to evaluate the texture complexity of the image to be processed based on the variance of the feature vector.

[0206] In an optional embodiment, the image region generation module includes:

[0207] The segmentation unit is used to slide the sliding window on the image to be processed to segment the image to be processed and obtain the preset number of image regions.

[0208] In two adjacent sliding processes, the sliding windows partially overlap.

[0209] In an optional embodiment, the object recognition results corresponding to each of the preset number of image regions include object location information corresponding to each of the preset number of image regions, and the device includes:

[0210] The original location information acquisition module is used to acquire the original location information of the upper left corner of each image region in the image to be processed.

[0211] The target location information generation module is used to generate target location information for each of the preset number of image regions in the image to be processed, based on the object location information corresponding to each of the preset number of image regions and the original location information.

[0212] In an optional embodiment, the object recognition result includes object category and object location information, and the combining module includes:

[0213] The category location combination unit is used to combine the object categories corresponding to each of the preset number of image regions to obtain the object category to which the object in the image to be processed belongs, and to combine the object location information corresponding to each of the preset number of image regions to obtain the object location information of the object in the image to be processed.

[0214] The object recognition result generation unit is used to generate the object recognition result corresponding to the image to be processed based on the object category to which the object in the image to be processed belongs and the object location information of the object in the image to be processed.

[0215] In an optional embodiment, the combined module includes:

[0216] The overlapping object recognition result determination unit is used to determine the overlapping object recognition result from all object recognition results corresponding to the preset number of image regions;

[0217] The deletion unit is used to delete the overlapping object recognition results from all object recognition results corresponding to the preset number of image regions to obtain candidate object recognition results;

[0218] The candidate combination unit is used to combine the candidate object recognition results to obtain the object recognition result corresponding to the image to be processed.

[0219] In an optional embodiment, the object recognition result generation module includes:

[0220] The first input unit is used to input the preset number of image regions to the visual encoder for visual feature conversion, so as to obtain the visual feature vectors corresponding to each of the preset number of image regions.

[0221] The second input unit is used to input the visual feature vectors corresponding to each of the preset number of image regions to the text decoder to generate a token sequence, thereby obtaining the token sequence corresponding to each of the preset number of image regions.

[0222] The conversion unit is used to convert the token sequences corresponding to each of the preset number of image regions into a format to obtain the object recognition results corresponding to each of the preset number of image regions.

[0223] In an optional embodiment, the device includes:

[0224] The sample texture complexity determination module is used to determine the sample texture complexity of the sample image.

[0225] A sample sliding window determination module is used to determine the sample sliding window corresponding to the sample image based on the sample texture complexity.

[0226] The sample image region generation module is used to segment the sample image based on the sample sliding window to obtain a preset number of sample image regions corresponding to the sample image;

[0227] The predicted object recognition result generation module is used to input the preset number of sample image regions, the sample images, and preset text prompt information into a preset multimodal large model for object recognition processing, and obtain the predicted object recognition results corresponding to each of the preset number of sample image regions;

[0228] The loss data generation module is used to determine loss data based on the predicted object identification result and the preset text prompt information;

[0229] The model generation module is used to adjust the model parameters of the preset multimodal large model according to the loss data until the preset multimodal large model converges, thereby obtaining the trained multimodal large model.

[0230] It should be noted that the device embodiments provided in this application are based on the same inventive concept as the method embodiments described above.

[0231] This application also provides an electronic device for image processing, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads and executes the at least one instruction or at least one program to implement the image processing method provided in any of the above embodiments.

[0232] This application also provides an electronic device for image processing, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads and executes the at least one instruction or at least one program to implement the image processing method provided in any of the above embodiments.

[0233] Embodiments of this application also provide a computer-readable storage medium that can be disposed in a terminal to store at least one instruction or at least one program for implementing an image processing method in the method embodiments. The at least one instruction or at least one program is loaded and executed by a processor to implement the image processing method provided in the above method embodiments.

[0234] Optionally, in the embodiments of this specification, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0235] The memory described in this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for functions, etc.; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0236] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method provided in the above-described method embodiments.

[0237] The image processing method provided in this application can be executed on a terminal, computer terminal, server, or similar computing device. Taking running on a server as an example... Figure 8 This is a hardware structure block diagram of a server according to an exemplary embodiment. For example... Figure 8 As shown, the server 400 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 410 (CPUs 410 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 430 for storing data, and one or more storage media 420 (e.g., one or more mass storage devices) for storing application programs 423 or data 422. The memory 430 and storage media 420 may be temporary or persistent storage. The program stored in the storage media 420 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 410 may be configured to communicate with the storage media 420 and execute the series of instruction operations stored in the storage media 420 on the server 400. Server 400 may also include one or more power supplies 460, one or more wired or wireless network interfaces 450, one or more input / output interfaces 440, and / or one or more operating systems 421, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0238] The input / output interface 440 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 400. In one example, the input / output interface 440 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 440 may be a radio frequency (RF) module for wireless communication with the Internet.

[0239] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 400 may also include... Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0240] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0241] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and server embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0242] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0243] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.< / answer> < / question> < / answer> < / question>

Claims

1. An image processing method, characterized by, The method comprises: determining the texture complexity of the to-be-processed image; the resolution of the to-be-processed image is greater than a preset resolution threshold; determining a sliding window corresponding to the to-be-processed image according to the texture complexity; segmenting the to-be-processed image based on the sliding window to obtain a preset number of image regions corresponding to the to-be-processed image; inputting the preset number of image regions into a multi-modal large model for object recognition processing to obtain object recognition results corresponding to the preset number of image regions respectively; wherein the multi-modal large model is obtained by training a preset multi-modal large model based on sample image regions and sample images, the sample image regions are obtained by segmenting the sample images based on sample sliding windows, and the sample sliding windows are determined based on the sample texture complexity of the sample images; combining the object recognition results corresponding to the preset number of image regions respectively to obtain object recognition results corresponding to the to-be-processed image.

2. The method of claim 1, wherein, The method comprises: obtaining a preset variance threshold; the preset variance threshold is determined based on the distribution of the variance of the texture complexity of the sample image; in the case that the texture complexity is greater than or equal to the preset variance threshold, determining that the sliding window corresponding to the to-be-processed image is a first sliding window, and in the case that the texture complexity is less than the preset variance threshold, determining that the sliding window corresponding to the to-be-processed image is a second sliding window; wherein the size of the first sliding window is smaller than the size of the second sliding window.

3. The method of claim 1, wherein, The method comprises: determining a preset direction, a preset frequency and a standard deviation; combining the preset direction and the preset frequency to obtain a preset number of Gaussian functions; convolving the to-be-processed image with each Gaussian function to obtain a response image corresponding to each Gaussian function; performing texture feature statistical processing on the response image corresponding to each Gaussian function to generate a texture feature corresponding to each Gaussian function; generating the texture complexity of the to-be-processed image according to the variance of the texture feature corresponding to each Gaussian function.

4. The method of claim 3, wherein, The method comprises: determining the mean and standard deviation of the response image corresponding to each Gaussian function to obtain the texture feature corresponding to each Gaussian function; The method comprises: combining the mean and standard deviation of the response image corresponding to each Gaussian function to obtain a feature vector; performing variance calculation on the mean and standard deviation of the response image corresponding to each Gaussian function, the mean of the feature vector and the number of Gaussian functions to obtain the variance of the feature vector; evaluating the texture complexity of the to-be-processed image according to the variance of the feature vector.

5. The method of claim 1, wherein, The method comprises: The sliding window is slid on the to-be-processed image based on the sliding window to segment the to-be-processed image, and the preset number of image regions are obtained. In the adjacent two sliding processes, the sliding windows partially overlap.

6. The method of claim 1, wherein, The object recognition result corresponding to each of the preset number of image regions includes object position information corresponding to each of the preset number of image regions. After the inputting of the preset number of image regions into the multi-modal large model for object recognition processing to obtain the object recognition result corresponding to each of the preset number of image regions, the method comprises: Obtaining original position information of a top-left corner of each image region in the to-be-processed image; Generating target position information of each of the preset number of image regions in the to-be-processed image according to the object position information corresponding to each of the preset number of image regions and the original position information.

7. The method of claim 1, wherein, The object recognition result includes object categories and object position information. The combining of the object recognition result corresponding to each of the preset number of image regions to obtain the object recognition result corresponding to the to-be-processed image comprises: Combining the object categories corresponding to each of the preset number of image regions to obtain an object category to which an object in the to-be-processed image belongs, and combining the object position information corresponding to each of the preset number of image regions to obtain object position information of the object in the to-be-processed image; Generating the object recognition result corresponding to the to-be-processed image according to the object category to which the object in the to-be-processed image belongs and the object position information of the object in the to-be-processed image.

8. The method of claim 1, wherein, The combining of the object recognition result corresponding to each of the preset number of image regions to obtain the object recognition result corresponding to the to-be-processed image comprises: Determining overlapping object recognition results from all object recognition results corresponding to the preset number of image regions; Deleting the overlapping object recognition results from all object recognition results corresponding to the preset number of image regions to obtain candidate object recognition results; Combining the candidate object recognition results to obtain the object recognition result corresponding to the to-be-processed image.

9. The method of claim 1, wherein, The multi-modal large model comprises a visual encoder and a text decoder. The inputting of the preset number of image regions into the multi-modal large model for object recognition processing to obtain the object recognition result corresponding to each of the preset number of image regions comprises: Inputting the preset number of image regions into the visual encoder for visual feature conversion to obtain a visual feature vector corresponding to each of the preset number of image regions; Inputting the visual feature vector corresponding to each of the preset number of image regions into the text decoder for token sequence generation to obtain a token sequence corresponding to each of the preset number of image regions; Converting the format of the token sequence corresponding to each of the preset number of image regions to obtain the object recognition result corresponding to each of the preset number of image regions.

10. The method according to any one of claims 1 to 9, characterized in that, The training process of the multi-modal large model comprises: Determining a sample texture complexity of a sample image; Determining a sample sliding window corresponding to the sample image according to the sample texture complexity; segment the sample image based on the sample sliding window to obtain a preset number of sample image regions corresponding to the sample image; input the preset number of sample image regions, the sample image, and preset text prompt information to a preset multi-modal large model for object recognition processing to obtain a prediction object recognition result corresponding to each of the preset number of sample image regions; determine loss data according to the prediction object recognition result and the preset text prompt information; adjust model parameters of the preset multi-modal large model according to the loss data until the preset multi-modal large model converges to obtain a trained multi-modal large model.

11. An image processing apparatus characterized by comprising: The apparatus comprises: a texture complexity determination module configured to determine a texture complexity of a to-be-processed image, wherein a resolution of the to-be-processed image is greater than a preset resolution threshold; a sliding window determination module configured to determine a sliding window corresponding to the to-be-processed image according to the texture complexity; an image region generation module configured to segment the to-be-processed image based on the sliding window to obtain a preset number of image regions corresponding to the to-be-processed image; an object recognition result generation module configured to input the preset number of image regions to a multi-modal large model for object recognition processing to obtain an object recognition result corresponding to each of the preset number of image regions, wherein the multi-modal large model is obtained by training a preset multi-modal large model based on sample image regions and sample images, the sample image regions are obtained by segmenting sample images based on sample sliding windows, and the sample sliding windows are obtained based on sample texture complexities of the sample images; a combination module configured to combine the object recognition results corresponding to the preset number of image regions to obtain an object recognition result corresponding to the to-be-processed image.

12. An electronic device for image processing, the electronic device comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 10.

Images