Multi-modal image generation system and method based on AI image analysis

Abstract

The invention discloses a multi-modal image generation system and method based on AI image analysis, and relates to the technical field of image processing, definition evaluation is performed based on pixel gray value data of each region of a to-be-processed image, and each key region of the to-be-processed image is determined according to a definition evaluation result; obtaining an image health response index based on the density data of each region of the to-be-processed image, the image recognition result accuracy data after each region of the to-be-processed image is missing, and the similarity data of each region of the to-be-processed image; and obtaining a processing demand response index based on the definition evaluation result and the image health response index of each key area of the to-be-processed image, performing corresponding operation on each key area of the to-be-processed image according to the processing demand response index, and performing corresponding definition processing operation on each key area of the to-be-processed image. The accuracy of information transmission in the network is ensured, a large number of computing resources are saved, the data storage space is reduced, and the response speed and stability of the whole system are improved.

Description

Technical Field

[0001] This application belongs to the field of image processing, specifically a multimodal image generation system and method based on AI image analysis. Background Technology

[0002] With the rapid popularization of the Internet, the sharing of digital images has become an important part of social networks. People can easily share information with each other through digital images. The development of digital images on the Internet has greatly improved the efficiency of information dissemination, and its influence is constantly growing. Digital images, as an important carrier of information transmission on the Internet, often suffer from blurring and significant noise due to limited image resolution, environmental interference, and image degradation during transmission. This severely affects the accuracy of information transmission over the network. Therefore, various image restoration methods are used to restore defective images. There are many ways to restore defective images, such as filtering and enhancement. In existing technologies, when performing image filtering operations on defective images, high-pass filtering or median filtering is applied to the entire defective image to improve its quality. However, this operation of restoring the entire defective image undoubtedly wastes a lot of computing resources, especially when the defective image is large. Furthermore, when processing defective images, intermediate data and results need to be stored. Processing the entire defective image greatly increases the storage space requirements, adds unnecessary data storage, and reduces the overall system response speed and stability. To address the problems raised in this background, this application designs a multimodal image generation system and method based on AI image analysis. Summary of the Invention

[0003] To address the aforementioned technical shortcomings, this application proposes a multimodal image generation system and method based on AI image analysis.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: This application provides a multimodal image generation method based on AI image analysis, which includes the following specific steps: S1. Obtain pixel grayscale data of each region of the image to be processed, density data of each region of the image to be processed, accuracy data of image recognition results after each region of the image to be processed is missing, and similarity data of each region of the image to be processed. S2. Based on the pixel grayscale value data of each region of the image to be processed, the image clarity of each region of the image to be processed is evaluated, and the key regions of the image to be processed are determined according to the image clarity evaluation results of each region of the image to be processed. S3. Based on the density data of each region of the image to be processed, the accuracy data of image recognition results after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed, the image health response index of each key region of the image to be processed is obtained. S4. Based on the image clarity assessment results and the image health response index of each key region of the image to be processed, obtain the processing demand response index of each key region of the image to be processed, and perform corresponding operations on each key region of the image to be processed according to the processing demand response index of each key region of the image to be processed.

[0005] It should be noted that, as a preferred technical solution for the multimodal image generation method based on AI image analysis, the specific steps of S1 are as follows: S11. Obtain the pixel grayscale value data of each region of the image to be processed by using the grayscale value data of each pixel in the image to be processed; S12. Obtain density data for each region of the image to be processed by using the pixel count data for each region of the image to be processed. S13. Obtain the image recognition accuracy data after each region of the image to be processed is missing from the database. S14. Calculate the structural similarity of each region of the image to be processed using similarity measurement to obtain the similarity data of each region of the image to be processed. S15. Store the collected data in the storage component for use in the analysis process.

[0006] It should be noted that, as a preferred technical solution for a multimodal image generation method based on AI image analysis, step S2 includes the following specific steps: S21. Import the pixel grayscale value data of each region of the image to be processed into the image sharpness evaluation value calculation formula to calculate the sharpness evaluation value corresponding to the i-th region of the image to be processed. The calculation formula for the image sharpness evaluation value corresponding to the i-th region of the image to be processed is as follows: Where i is the number corresponding to each region block in the image to be processed, and i is any one of 1 to N. Let the horizontal Sobel operator convolution kernel be the i-th region of the image to be processed. Let the Sobel operator convolution kernel be the vertical direction of the i-th region of the image to be processed. Let be the grayscale matrix corresponding to the i-th region of the image to be processed. The gradient magnitude is a set reference value. It should be noted that in this formula... The gradient magnitude refers to the gradient magnitude at the coordinates (x, y) of the pixel point in the i-th region of the image to be processed. The gradient magnitude directly reflects the maximum rate of change of the gray value of a pixel at a certain point in each region of the image to be processed. The larger the gradient magnitude, the more drastic the change at that point, and the smaller the gradient magnitude, the more gradual the change. The higher the overall gradient magnitude of the image to be processed, the better the clarity of the image to be processed. S22. Compare the image sharpness evaluation value corresponding to each region of the image to be processed with the set image sharpness evaluation value threshold. If the image sharpness evaluation value corresponding to a certain region of the image to be processed is less than or equal to the set image sharpness evaluation value threshold, mark it as a key region and proceed to S3; if the image sharpness evaluation value corresponding to a certain region of the image to be processed is greater than the set image sharpness evaluation value threshold, then end.

[0007] It should be noted that, as a preferred technical solution for the multimodal image generation method based on AI image analysis, the specific steps of S3 are as follows: S31. Obtain the density data of each key region of the image to be processed, the accuracy data of the image recognition result after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed from the density data of each region of the image to be processed, the accuracy data of the image recognition result after each key region of the image to be processed is missing, and the similarity data of each region of the image to be processed. S32. The importance evaluation value of each key region of the image to be processed is obtained from the density data of each key region of the image to be processed and the accuracy data of the image recognition results after the loss of each key region of the image to be processed. S33. Obtain the image impact assessment value of each key region of the image to be processed from the similarity data of each key region and the connected regions of the image to be processed. S34. The image health response index of each key region of the image to be processed is obtained by weighting and summing the image importance assessment value and image impact assessment value.

[0008] It should be noted that, as a preferred technical solution for the multimodal image generation method based on AI image analysis, the specific steps of S32 are as follows: The density data of each key region of the image to be processed and the accuracy data of image recognition results after the loss of each key region of the image to be processed are imported into the formula for calculating the image importance evaluation value of each key region of the image to be processed. The formula for calculating the image importance evaluation value of the j-th key region of the image to be processed is as follows: Where j is the number corresponding to each key region in the image to be processed, and j is any one of 1 to M. Let be the number corresponding to the grayscale value of each pixel in the j-th key region of the image to be processed. It can be any one of 1 to W. For the j-th key region of the image to be processed Each pixel's grayscale value The key region of the image to be processed The probability of a pixel grayscale value appearing in the key region, H is the reference value of image entropy in the key region, and a is the weight of the image entropy value. Let be the accuracy fluctuation value of image recognition results after the j-th key region of the image to be processed is missing, and Er be the reference accuracy fluctuation value. It should be noted that in this formula... Part of the method measures the amount of information and the degree of disorder in each key region of the image under processing by the gray values ​​of each pixel and the probability of occurrence of each pixel gray value, that is, quantifies the complexity and detail richness of each key region of the image under processing; in this formula The accuracy fluctuation of image recognition results was evaluated by assessing the importance of each key region of the image to be processed in terms of recognition accuracy.

[0009] It should be noted that, as a preferred technical solution for the multimodal image generation method based on AI image analysis, the specific steps of S33 are as follows: The similarity data between each key region and adjacent regions of the image to be processed is imported into the formula for calculating the image impact assessment value of the key regions of the image to be processed, whereby the formula for calculating the image impact assessment value of the j-th key region of the image to be processed is: Where k is the number corresponding to each connected region of the key region in the image to be processed, and k is any one of 1 to Q. The element is a component in the similarity correlation matrix between the j-th key region and the k-th connected region of the image to be processed. This represents the maximum sum of similarities between each key region and its adjacent regions in the image to be processed. It should be noted that... This formula represents the similarity between key regions and adjacent regions in the image to be processed. Part of the method measures the importance of each key region in the image to be processed by the sum of the similarity between the key region and all connected regions in the image to be processed, and evaluates the criticality of each key region in the image to be processed by calculating the similarity between each key region and other connected regions in the image to be processed.

[0010] It should be noted that, as a preferred technical solution for the multimodal image generation method based on AI image analysis, the specific steps of S4 are as follows: S41. Obtain the sharpness assessment value corresponding to each key region of the image to be processed. Import the sharpness assessment value of each key region of the image to be processed and the image health response index into the processing demand response index calculation formula of the image to be processed to calculate the processing demand response index of each key region of the image to be processed. The calculation formula for the processing demand response index of the j-th key region of the image to be processed is as follows: ,in, Let be the image health response index corresponding to the j-th key region of the image to be processed. It should be noted that this formula combines the sharpness assessment value of the key region of the image to be processed and the image health response index to obtain the processing demand response index of each key region of the image to be processed, thereby improving the accuracy of the processing demand response index of each key region of the image to be processed. S42. Compare the processing demand response index of each key area of ​​the image to be processed with the set processing demand response index threshold. If the processing demand response index of a key area of ​​the image to be processed is less than or equal to the set processing demand response index threshold, then the key area of ​​the image to be processed is sharpened. If the processing demand response index of a key area of ​​the image to be processed is greater than the set processing demand response index threshold, then the key area of ​​the image to be processed is not sharpened.

[0011] The multimodal image generation system based on AI image analysis is implemented based on the above-mentioned multimodal image generation method based on AI image analysis. Specifically, it includes an image data acquisition module, an image sharpness assessment module, a health response index analysis module, and an image sharpening processing module. The image data acquisition module is used to acquire pixel grayscale value data of each region of the image to be processed, density data of each region of the image to be processed, accuracy data of image recognition results after each region of the image to be processed is missing, and similarity data of each region of the image to be processed. The image sharpness evaluation module is used to evaluate the image sharpness of each region of the image to be processed based on the pixel grayscale value data of each region of the image to be processed, and to determine the key regions of the image to be processed based on the image sharpness evaluation results of each region of the image to be processed. The health response index analysis module is used to obtain the image health response index of each key region of the image to be processed based on the density data of each region of the image to be processed, the accuracy data of image recognition results after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed. The image sharpening processing module is used to obtain the processing demand response index of each key area of ​​the image to be processed based on the image sharpness assessment result and the image health response index of each key area of ​​the image to be processed, and to perform corresponding operations on each key area of ​​the image to be processed according to the processing demand response index of each key area of ​​the image to be processed.

[0012] An electronic device includes: a processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor executes the aforementioned multimodal image generation method based on AI image analysis by calling the computer program stored in the memory.

[0013] A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned multimodal image generation method based on AI image analysis.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention acquires pixel grayscale value data, density data, image recognition accuracy data after missing regions of the image to be processed, and similarity data of each region of the image to be processed; it evaluates the image clarity of each region of the image to be processed based on the pixel grayscale value data, and determines key regions of the image to be processed based on the image clarity evaluation results; it obtains the image health response index of each key region of the image to be processed based on the density data, image recognition accuracy data after missing regions of the image to be processed, and similarity data of each region of the image to be processed; it obtains the processing demand response index of each key region of the image to be processed based on the image clarity evaluation results and the image health response index of each key region of the image to be processed; and it performs corresponding operations on each key region of the image to be processed based on the processing demand response index of each key region of the image to be processed, performing corresponding clarity processing operations on each key region of the image to be processed, ensuring the accuracy of information transmission in the network, saving a lot of computing resources, reducing data storage space, and improving the overall system response speed and stability. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall process of the multimodal image generation method based on AI image analysis in this application.

[0016] Figure 2 This is a schematic diagram of step S3 of the multimodal image generation method based on AI image analysis in this application.

[0017] Figure 3 This is a schematic diagram of the overall framework of the multimodal image generation system based on AI image analysis in this application.

[0018] Figure 4 This is a schematic diagram illustrating the process of obtaining the image health response index for the multimodal image generation method based on AI image analysis in this application. Detailed Implementation

[0019] To better understand this application, various aspects of this application will be described in more detail with reference to the accompanying drawings.

[0020] To address the technical problems raised in the background art, this application provides a preferred embodiment: The specific content of this embodiment is as follows: like Figure 1As shown, the multimodal image generation method based on AI image analysis includes the following specific steps: S1. Obtain pixel grayscale data of each region of the image to be processed, density data of each region of the image to be processed, accuracy data of image recognition results after each region of the image to be processed is missing, and similarity data of each region of the image to be processed. In this embodiment, the specific steps of S1 are as follows: S11. Obtain the pixel grayscale value data of each region of the image to be processed by using the grayscale value data of each pixel in the image to be processed; S12. Obtain density data for each region of the image to be processed by using the pixel count data for each region of the image to be processed. S13. Obtain the image recognition accuracy data after each region of the image to be processed is missing from the database. S14. Calculate the structural similarity of each region of the image to be processed using similarity measurement to obtain the similarity data of each region of the image to be processed. S15. Store the collected data in the storage component for use in the analysis process.

[0021] In one implementation of the present invention, the pixel grayscale value data of each region of the image to be processed is obtained by acquiring the grayscale value data of each pixel in the image to be processed. The gradient magnitude of the pixel coordinates of each region of the image to be processed is analyzed to directly reflect the maximum rate of change of the pixel grayscale value of a certain point in each region of the image to be processed. The density data of each region of the image to be processed is obtained by acquiring the pixel quantity data of each region of the image to be processed. The complexity and detail richness of each key region of the image to be processed are analyzed and quantified. The image recognition accuracy data after each region of the image to be processed is obtained by acquiring the data from the database. The importance of each key region of the image to be processed to the recognition accuracy of the image to be processed is analyzed. The structural similarity of each region of the image to be processed is calculated by similarity measurement to obtain the similarity data of each region of the image to be processed. The similarity between each key region and other connected regions in the image to be processed is analyzed to evaluate the criticality of each key region in the image to be processed.

[0022] S2. Based on the pixel grayscale value data of each region of the image to be processed, the image clarity of each region of the image to be processed is evaluated, and the key regions of the image to be processed are determined according to the image clarity evaluation results of each region of the image to be processed. In this embodiment, S2 includes the following specific steps: S21. Import the pixel grayscale value data of each region of the image to be processed into the image sharpness evaluation value calculation formula to calculate the sharpness evaluation value corresponding to the i-th region of the image to be processed. The calculation formula for the image sharpness evaluation value corresponding to the i-th region of the image to be processed is as follows: Where i is the number corresponding to each region block in the image to be processed, and i is any one of 1 to N. Let the horizontal Sobel operator convolution kernel be the i-th region of the image to be processed. Let the Sobel operator convolution kernel be the vertical direction of the i-th region of the image to be processed. Let be the grayscale matrix corresponding to the i-th region of the image to be processed. The gradient magnitude is a set reference value. It should be noted that in this formula... The gradient magnitude refers to the gradient magnitude at the coordinates (x, y) of the pixel point in the i-th region of the image to be processed. The gradient magnitude directly reflects the maximum rate of change of the gray value of a pixel at a certain point in each region of the image to be processed. The larger the gradient magnitude, the more drastic the change at that point, and the smaller the gradient magnitude, the more gradual the change. The higher the overall gradient magnitude of the image to be processed, the better the clarity of the image. For example, when the image to be processed has sharp edges such as object boundaries or text lines, the gradient magnitude changes as follows: the gradient magnitude increases sharply, forming a steep peak. When the image to be processed has soft transitions such as shadow gradients or blurred areas, the gradient magnitude changes as follows: the gradient magnitude is gentle and there are no significant peaks. S22. Compare the image sharpness evaluation value corresponding to each region of the image to be processed with the set image sharpness evaluation value threshold. If the image sharpness evaluation value corresponding to a certain region of the image to be processed is less than or equal to the set image sharpness evaluation value threshold, mark it as a key region and proceed to S3; if the image sharpness evaluation value corresponding to a certain region of the image to be processed is greater than the set image sharpness evaluation value threshold, then end.

[0023] S3. Based on the density data of each region of the image to be processed, the accuracy data of image recognition results after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed, the image health response index of each key region of the image to be processed is obtained. like Figure 2 As shown, in this embodiment, the specific steps of S3 are as follows: S31. Obtain the density data of each key region of the image to be processed, the accuracy data of the image recognition result after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed from the density data of each region of the image to be processed, the accuracy data of the image recognition result after each key region of the image to be processed is missing, and the similarity data of each region of the image to be processed. S32. The importance evaluation value of each key region of the image to be processed is obtained from the density data of each key region of the image to be processed and the accuracy data of the image recognition results after the loss of each key region of the image to be processed. In this embodiment, the specific steps of S32 are as follows: The density data of each key region of the image to be processed and the accuracy data of image recognition results after the loss of each key region of the image to be processed are imported into the formula for calculating the image importance evaluation value of each key region of the image to be processed. The formula for calculating the image importance evaluation value of the j-th key region of the image to be processed is as follows: Where j is the number corresponding to each key region in the image to be processed, and j is any one of 1 to M. Let be the number corresponding to the grayscale value of each pixel in the j-th key region of the image to be processed. It can be any one of 1 to W. For the j-th key region of the image to be processed Each pixel's grayscale value The key region of the image to be processed The probability of a pixel grayscale value appearing in the key region, H is the reference value of image entropy in the key region, and a is the weight of the image entropy value. Let be the accuracy fluctuation value of image recognition results after the j-th key region of the image to be processed is missing, and Er be the reference accuracy fluctuation value. It should be noted that in this formula... Part of the method measures the amount of information and the degree of disorder in each key region of the image under processing by the gray values ​​of each pixel and the probability of occurrence of each pixel gray value, that is, quantifies the complexity and detail richness of each key region of the image under processing; in this formula The accuracy fluctuation of image recognition results after the loss of key regions in the image to be processed is evaluated. The results reflect the importance of each key region in the image to the recognition accuracy. For example, the influence of pixel gray values ​​on the image importance evaluation value of each key region in the image to be processed is illustrated. If the gray value distribution in a key region of the image to be processed is relatively concentrated, it means that the image entropy value of the key region of the image to be processed is low, that is, the information content of the key region of the image to be processed is small, and the image importance evaluation value of the key region of the image to be processed decreases. If the gray value distribution in a key region of the image to be processed is relatively uniform, it means that the image entropy value of the key region of the image to be processed is high, that is, the information content of the key region of the image to be processed is large, and the image importance evaluation value of the key region of the image to be processed increases.

[0024] S33. Obtain the image impact assessment value of each key region of the image to be processed from the similarity data of each key region and the connected regions of the image to be processed. In this embodiment, the specific step of S33 is as follows: The similarity data between each key region and its adjacent regions in the image to be processed is imported into the formula for calculating the image impact assessment value of each key region in the image to be processed. The formula for calculating the image impact assessment value of the j-th key region in the image to be processed is: Where k is the number corresponding to each connected region of the key region in the image to be processed, and k is any one of 1 to Q. The element is a component in the similarity correlation matrix between the j-th key region and the k-th connected region of the image to be processed. This represents the maximum sum of similarities between each key region and its adjacent regions in the image to be processed. It should be noted that... This formula represents the similarity between key regions and adjacent regions in the image to be processed. Part of the method measures the importance of each key region in the image to be processed by summing the similarity between the key region and all its connected regions. It evaluates the criticality of each key region in the image to be processed by calculating the similarity between each key region and other connected regions. For example, to illustrate the impact of the similarity between the key region and its connected regions on the image impact assessment value of each key region in the image to be processed, in an image to be processed containing people, background and objects, the people region and the object region are the key regions of the image to be processed. If the sum of the similarity between the people region and its corresponding connected regions is greater than the sum of the similarity between the object region and its corresponding connected regions, then the image impact assessment value of the people region is greater than the image impact assessment value of the object region.

[0025] S34, such as Figure 4 As shown, the image health response index of each key region of the image to be processed is obtained by weighting and summing the image importance assessment value and image impact assessment value of each key region of the image to be processed.

[0026] S4. Based on the image clarity assessment results and the image health response index of each key region of the image to be processed, obtain the processing demand response index of each key region of the image to be processed, and perform corresponding operations on each key region of the image to be processed according to the processing demand response index of each key region of the image to be processed.

[0027] In this embodiment, the specific steps of S4 are as follows: S41. Obtain the sharpness assessment value corresponding to each key region of the image to be processed. Import the sharpness assessment value of each key region of the image to be processed and the image health response index into the processing demand response index calculation formula of the image to be processed to calculate the processing demand response index of each key region of the image to be processed. The calculation formula for the processing demand response index of the j-th key region of the image to be processed is as follows: ,in, Let be the image health response index corresponding to the j-th key region of the image to be processed. It should be noted that this formula combines the sharpness assessment value of the key region of the image to be processed and the image health response index to obtain the processing demand response index of each key region of the image to be processed, which improves the accuracy of the processing demand response index of each key region of the image to be processed. For example, to illustrate the relationship between the sharpness assessment value of each key region of the image to be processed and the processing demand response index of each key region of the image to be processed, the smaller the sharpness assessment value of the key region of the image to be processed, the more the key region of the image to be processed needs to be processed, that is, the larger the processing demand response index of each key region of the image to be processed.

[0028] S42. Compare the processing demand response index of each key region of the image to be processed with the set processing demand response index threshold. If the processing demand response index of a key region of the image to be processed is less than or equal to the set processing demand response index threshold, then the key region of the image to be processed is sharpened. If the processing demand response index of a key region of the image to be processed is greater than the set processing demand response index threshold, then the key region of the image to be processed is not sharpened. For example, the methods of sharpening a key region of the image to be processed are illustrated, including but not limited to: dehazing, noise reduction, and super-resolution.

[0029] It should be noted that the parameters set in this embodiment (such as weights and thresholds) need to be set by those skilled in the art based on relevant experiments. The specific experimental method is as follows: obtain the pixel grayscale value data of each region of the image to be processed, the density data of each region of the image to be processed, the accuracy data of image recognition results after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed. Substitute these data into the steps of this embodiment to calculate the processing demand response index of each key region of the image to be processed. Obtain the processing demand response index of each key region of the image to be processed. Import the calculation results of the processing demand response index of each key region of the image to be processed and the results of the required sharpening operation of each key region of the image to be processed into the fitting software (MATLAB) to continuously fit the data. Output the processing demand response index of each key region of the image to be processed, which conforms to the set parameters (such as weights and thresholds) of the required sharpening operation of each key region of the image to be processed.

[0030] Based on the above implementation, this embodiment has the following advantages over the prior art: This embodiment acquires pixel grayscale value data of each region of the image to be processed, density data of each region of the image to be processed, accuracy data of image recognition results after each region of the image to be processed is missing, and similarity data of each region of the image to be processed; it evaluates the image clarity of each region of the image to be processed based on the pixel grayscale value data of each region of the image to be processed, and determines each key region of the image to be processed based on the image clarity evaluation results of each region of the image to be processed; it obtains the image health response index of each key region of the image to be processed based on the density data of each region of the image to be processed, the accuracy data of image recognition results after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed; it obtains the processing demand response index of each key region of the image to be processed based on the image clarity evaluation results and the image health response index of each key region of the image to be processed, and performs corresponding operations on each key region of the image to be processed according to the processing demand response index of each key region of the image to be processed, and performs corresponding clarity processing operations on each key region of the image to be processed, ensuring the accuracy of information transmission in the network, saving a lot of computing resources, reducing data storage space, and improving the overall system response speed and stability.

[0031] like Figure 3 As shown, this embodiment also provides a multimodal image generation system based on AI image analysis, which is implemented based on the above-mentioned multimodal image generation method based on AI image analysis. Specifically, it includes an image data acquisition module, an image sharpness evaluation module, a health response index analysis module, and an image sharpening processing module. The image data acquisition module is used to acquire pixel grayscale value data of each region of the image to be processed, density data of each region of the image to be processed, accuracy data of image recognition results after each region of the image to be processed is missing, and similarity data of each region of the image to be processed. The image sharpness evaluation module is used to evaluate the image sharpness of each region of the image to be processed based on the pixel grayscale value data of each region of the image to be processed. The system estimates key regions of the image to be processed based on the image clarity assessment results of each region. The health response index analysis module calculates the image health response index of each key region of the image to be processed based on the density data of each region, the accuracy data of image recognition results after the missing regions of each region, and the similarity data of each region. The image sharpening processing module calculates the processing demand response index of each key region of the image to be processed based on the image clarity assessment results and the image health response index of each key region, and performs corresponding operations on each key region of the image to be processed according to the processing demand response index.

[0032] The specific steps for implementing the corresponding functions of each unit module in the multimodal image generation system based on AI image analysis of this application can be referred to the steps in the embodiments of the multimodal image generation method based on AI image analysis above, and will not be repeated here.

[0033] This embodiment also provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor executes the aforementioned multimodal image generation method based on AI image analysis by calling the computer program stored in the memory.

[0034] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory 310 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the AI ​​image analysis-based multimodal image generation method provided in the above embodiments. The data storage area may store data involved in the AI ​​image analysis-based multimodal image generation method provided in the above embodiments.

[0035] The processor may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data as described in this application. The processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit this.

[0036] It may also include a communication bus, which can include a pathway for transmitting information between the aforementioned components. The communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into address bus, data bus, control bus, etc.

[0037] This embodiment also proposes a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned multimodal image generation method based on AI image analysis.

[0038] For example, computer-readable storage media can be read-only memory, random access memory, read-only optical disc, magnetic tape, floppy disk, and optical data storage devices.

[0039] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired network and / or wireless network. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives (SSDs).

[0040] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0041] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions claimed in this application.

Claims

1. A multimodal image generation method based on AI image analysis, characterized in that, include: S1. Obtain pixel grayscale data of each region of the image to be processed, density data of each region of the image to be processed, accuracy data of image recognition results after each region of the image to be processed is missing, and similarity data of each region of the image to be processed. S2. Based on the pixel grayscale value data of each region of the image to be processed, the image clarity of each region of the image to be processed is evaluated, and the key regions of the image to be processed are determined according to the image clarity evaluation results of each region of the image to be processed. S3. Based on the density data of each region of the image to be processed, the accuracy data of image recognition results after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed, the image health response index of each key region of the image to be processed is obtained. S4. Based on the image clarity assessment results and the image health response index of each key region of the image to be processed, obtain the processing demand response index of each key region of the image to be processed, and perform corresponding operations on each key region of the image to be processed according to the processing demand response index of each key region of the image to be processed.

2. The multimodal image generation method based on AI image analysis as described in claim 1, characterized in that, S2 includes the following specific steps: S21. Import the pixel grayscale value data of each region of the image to be processed into the image sharpness evaluation value calculation formula to calculate the sharpness evaluation value corresponding to the i-th region of the image to be processed. The calculation formula for the image sharpness evaluation value corresponding to the i-th region of the image to be processed is as follows: Where i is the number corresponding to each region block in the image to be processed, and i is any one of 1 to N. Let the horizontal Sobel operator convolution kernel be the i-th region of the image to be processed. Let the Sobel operator convolution kernel be the vertical direction of the i-th region of the image to be processed. Let be the grayscale matrix corresponding to the i-th region of the image to be processed. This is the set gradient magnitude reference value; S22. Compare the image sharpness evaluation value corresponding to each region of the image to be processed with the set image sharpness evaluation value threshold. If the image sharpness evaluation value corresponding to a certain region of the image to be processed is less than or equal to the set image sharpness evaluation value threshold, mark it as a key region and proceed to S3; if the image sharpness evaluation value corresponding to a certain region of the image to be processed is greater than the set image sharpness evaluation value threshold, then end.

3. The multimodal image generation method based on AI image analysis as described in claim 2, characterized in that, The specific steps of S3 are as follows: S31. Obtain the density data of each key region of the image to be processed, the accuracy data of the image recognition result after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed from the density data of each region of the image to be processed, the accuracy data of the image recognition result after each key region of the image to be processed is missing, and the similarity data of each region of the image to be processed. S32. The importance evaluation value of each key region of the image to be processed is obtained from the density data of each key region of the image to be processed and the accuracy data of the image recognition results after the loss of each key region of the image to be processed. S33. Obtain the image impact assessment value of each key region of the image to be processed from the similarity data of each key region and the connected regions of the image to be processed. S34. The image health response index of each key area of ​​the image to be processed is obtained by weighting and summing the image importance assessment value and image impact assessment value.

4. The multimodal image generation method based on AI image analysis as described in claim 3, characterized in that, The specific steps of S32 are as follows: The density data of each key region of the image to be processed and the accuracy data of image recognition results after the loss of each key region of the image to be processed are imported into the formula for calculating the image importance evaluation value of each key region of the image to be processed. The formula for calculating the image importance evaluation value of the j-th key region of the image to be processed is as follows: Where j is the number corresponding to each key region in the image to be processed, and j is any one of 1 to M. Let be the number corresponding to the grayscale value of each pixel in the j-th key region of the image to be processed. It can be any one of 1 to W. For the j-th key region of the image to be processed Each pixel's grayscale value The key region of the image to be processed The probability of a pixel grayscale value appearing in the key region, H is the reference value of image entropy in the key region, and a is the weight of the image entropy value. Er is the accuracy fluctuation value of the image recognition result after the j-th key region of the image to be processed is missing, and Er is the reference accuracy fluctuation value.

5. The multimodal image generation method based on AI image analysis as described in claim 4, characterized in that, The specific steps of S33 are as follows: The similarity data between each key region and its adjacent regions in the image to be processed is imported into the formula for calculating the image impact assessment value of each key region in the image to be processed. The formula for calculating the image impact assessment value of the j-th key region in the image to be processed is as follows: Where k is the number corresponding to each connected region of the key region in the image to be processed, and k is any one of 1 to Q. The element is a component in the similarity correlation matrix between the j-th key region and the k-th connected region of the image to be processed. This represents the maximum sum of similarities between each key region and its adjacent regions in the image to be processed.

6. The multimodal image generation method based on AI image analysis as described in claim 5, characterized in that, The specific steps of S4 are as follows: S41. Obtain the sharpness assessment value corresponding to each key region of the image to be processed. Import the sharpness assessment value of each key region of the image to be processed and the image health response index into the processing demand response index calculation formula of the image to be processed to calculate the processing demand response index of each key region of the image to be processed. The calculation formula for the processing demand response index of the j-th key region of the image to be processed is as follows: ,in, Let be the image health response index corresponding to the j-th key region of the image to be processed; S42. Compare the processing demand response index of each key area of ​​the image to be processed with the set processing demand response index threshold. If the processing demand response index of a key area of ​​the image to be processed is less than or equal to the set processing demand response index threshold, then the key area of ​​the image to be processed is sharpened. If the processing demand response index of a key area of ​​the image to be processed is greater than the set processing demand response index threshold, then the key area of ​​the image to be processed is not sharpened.

7. A multimodal image generation system based on AI image analysis, implemented based on the multimodal image generation method based on AI image analysis according to any one of claims 1-6, characterized in that, Specifically, it includes an image data acquisition module, an image sharpness assessment module, a health response index analysis module, and an image sharpening processing module. The image data acquisition module is used to acquire pixel grayscale value data of each region of the image to be processed, density data of each region of the image to be processed, accuracy data of image recognition results after each region of the image to be processed is missing, and similarity data of each region of the image to be processed. The image sharpness evaluation module is used to evaluate the image sharpness of each region of the image to be processed based on the pixel grayscale value data of each region of the image to be processed, and to determine the key regions of the image to be processed based on the image sharpness evaluation results of each region of the image to be processed. The health response index analysis module is used to obtain the image health response index of each key region of the image to be processed based on the density data of each region of the image to be processed, the accuracy data of image recognition results after each region of the image to be processed is missing, and the similarity data of each region of the image to be processed. The image sharpening processing module is used to obtain the processing demand response index of each key area of ​​the image to be processed based on the image sharpness assessment result and the image health response index of each key area of ​​the image to be processed, and to perform corresponding operations on each key area of ​​the image to be processed according to the processing demand response index of each key area of ​​the image to be processed.

8. An electronic device, comprising: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor is characterized in that it executes the multimodal image generation method based on AI image analysis as described in any one of claims 1-6 by calling a computer program stored in the memory.

9. A computer-readable storage medium, characterized in that, The system stores instructions that, when executed on a computer, cause the computer to perform the multimodal image generation method based on AI image analysis as described in any one of claims 1-6.

Images