An intelligent image processing method and system based on AI

Abstract

The present invention relates to the field of AI technology, and discloses an AI-based intelligent image processing method and system, the method comprising the following steps: a user provides input data, the input data includes original pictures, text descriptions and multimodal information; a pre-trained target detection algorithm is used to batch process the object recognition in all original pictures; the objects recognized in each picture are classified and marked to which they belong; a set of standardized features of each type of object is extracted; for each object in each picture, the standardized features are matched with the keywords in the user's text description. The present invention accurately understands the user's intentions and preferences by introducing multimodal information and extracting features from the original pictures provided by the user. A comprehensive scoring system based on feature similarity, keyword matching and multimodal influence enables important objects to be given priority in the generation process, ensuring that the generation results are more in line with user expectations.

Description

Technical Field

[0001] The present invention relates to the field of AI, and more specifically, to an AI-based intelligent image processing method and system. Background Art

[0002] In existing AI generation systems, the technology of AI generated images mainly relies on machine learning, especially the generation model in deep learning.

[0003] Existing AI generation technologies typically rely on a single type of input (such as text or pictures). Existing AI generation models may produce overly random results or fail to control the generated content well. At the same time, existing AI generation systems find it difficult to effectively evaluate the importance of each object in the picture, causing the generation results to deviate from user expectations. Summary of the invention

[0004] The present invention provides an AI-based intelligent image processing method and system to solve technical problems in related technologies.

[0005] The present invention provides an AI-based intelligent image processing method, comprising the following steps:

[0006] S100, preparing an original picture set and multimodal description: a user provides input data, the input data includes original pictures, text descriptions and multimodal information;

[0007] S200, Batch Object Detection, Classification and Feature Extraction: Use pre-trained object detection algorithms to batch process object recognition in all original images; classify the objects recognized in each image and label them as belonging to the same category; extract a set of standardized features for each type of object;

[0008] S300, intelligent sorting mechanism: for each object in each picture, a comprehensive score is obtained based on its standardized features and the keyword matching degree in the user's text description, combined with multimodal influence factors;

[0009] Sort the objects in all images according to their comprehensive scores and generate a sorted object list;

[0010] S400, initial batch generation of image groups: using a personalized generation model for a specific type or field, inputting a sorted list of objects and their priorities, and multimodal information as auxiliary input, to generate a set of new images that meet the requirements for each original image;

[0011] S500, for the new images that meet the requirements after processing, a second selection is performed by calculating the similarity score, and a batch of images that are closest to the expected effect are selected to form a new candidate set;

[0012] S600, feature analysis and parameter adjustment: in-depth feature analysis of the candidate set, determine the content that meets the user's expectations and the content that needs to be improved, and update the parameters in the personalized generation model;

[0013] S700, batch generation of final destination images: using the updated personalized generation model to run again, outputting the final destination image for each original image, and outputting the final generated image set;

[0014] S800, evaluation and feedback loop: the generated picture set is displayed to the user. If the user feedback is satisfactory, the generated picture is the target picture; if the user feedback is dissatisfied, the customer feedback content is integrated into the text description, and then return to step S100.

[0015] Furthermore, multimodal information includes audio and video.

[0016] Furthermore, the standardized features include color distribution, texture, shape, size, and position.

[0017] Furthermore, the personalized generation model includes a generator and the discriminator , generator is the random noise vector and condition information As input, the output generated image is ; Discriminator Is to receive real or generated images and condition information , output a probability value , indicating the given condition Next picture is the true probability.

[0018] Furthermore, for the conditional generative adversarial network, the following loss function is defined:

[0019] ;

[0020] ;

[0021] ;

[0022] in Represents random noise The distribution of is usually the standard normal distribution; Represents intermediate data; Indicates conditional information The conditional information includes text description, category label or other forms of information; Represents the actual data distribution.

[0023] Furthermore, the personalized generation model also requires parameter updates: including generator updates and discriminator updates;

[0024] Generator updates are minimizations of ,maximize:

[0025] ;

[0026] The discriminator update is to minimize , while maximizing:

[0027] ;

[0028] ;

[0029] in Represents a random noise vector, which comes from a standard normal distribution ; Representation of conditional information, including embedded vectors of text descriptions, classification labels, audio features, and video frame features; Represents a generated image or a real image.

[0030] Furthermore, the similarity score formula in the secondary selection is as follows:

[0031] ;

[0032] in Generate pictures The data characteristics, is a function to measure similarity, is the jth image generated after processing.

[0033] Furthermore, the user's expectations are integrated into the text description. Updated to , if there is additional multimodal information, update the condition information For new condition information .

[0034] The present invention further proposes an AI-based intelligent image processing system, which is used to perform one or more steps in the aforementioned AI-based intelligent image processing method, including:

[0035] User input and data preparation module: collect and preprocess user input data;

[0036] Batch object detection and classification module: identifies, classifies, and extracts features of objects in all original images;

[0037] Intelligent sorting module: sort objects according to object features, text matching and multimodal influence;

[0038] Initial generation module: Generate a set of new images for each original image based on the sorted object list and multimodal information;

[0039] Secondary selection and candidate set formation module: select the pictures closest to the expected effect from the initially generated pictures to form a candidate set;

[0040] Feature analysis and parameter adjustment module: conducts in-depth analysis of the candidate set to determine the direction of improvement and adjust the parameters of the generated model;

[0041] Final generation module: Use the updated model to output the final destination image for each original image;

[0042] Evaluation and feedback loop module: Show the generated images to users, get feedback, and further optimize the results as needed.

[0043] The present invention also proposes a storage medium storing non-temporary computer-readable instructions for executing one or more steps in the aforementioned AI-based intelligent image processing method.

[0044] The beneficial effects of the present invention are:

[0045] By introducing multimodal information (such as text, audio, and video) and extracting features from the original images provided by the user, the system can more accurately understand the user's intentions and preferences. The comprehensive scoring system based on feature similarity, keyword matching, and multimodal influence enables important objects to be given priority in the generation process, ensuring that the generation results are more in line with user expectations. The entire process is designed to effectively process a large number of input images and is suitable for application scenarios that require rapid generation of multiple high-quality images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of an AI-based intelligent image processing method of the present invention;

[0047] Figure 2 It is a structural block diagram of an AI-based intelligent image processing system of the present invention;

[0048] Figure 3 It is an execution flow chart of an AI-based intelligent image processing system of the present invention.

[0049] In the figure: 101, user input and data preparation module; 102, batch target detection and classification module; 103, intelligent sorting module; 104, initial generation module; 105, secondary selection and candidate set formation module; 106, feature analysis and parameter adjustment module; 107, final generation module; 108, evaluation and feedback loop module. DETAILED DESCRIPTION

[0050] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0051] like Figure 1 As shown, an AI-based intelligent image processing method includes the following steps:

[0052] S100, prepare original image set and multimodal description:

[0053] The user provides input data, which includes a set of original images similar to the target image. ,in is the number of pictures;

[0054] Users describe their needs in text using natural language , and can also provide additional multimodal information, such as audio ,video wait;

[0055] S200, batch object detection, classification and feature extraction:

[0056] Use a pre-trained object detection algorithm to batch process object recognition in all original images;

[0057] Classify the identified objects in each image and label them as belonging to the category (e.g. people, buildings, natural landscapes, etc.);

[0058] Extract a set of standardized features for each class of objects , including color distribution, texture, shape, size, position, etc.;

[0059] In one embodiment of the present invention, the original image is preprocessed (such as resizing and normalizing) by feature engineering to extract object features. , and extract features from multimodal information such as text, audio, and video;

[0060] Output a list of all objects in the image, their categories and features ,in Indicates the image index. Represents the object index, each Contains classification labels and feature vectors ;

[0061] In one embodiment of the present invention, the pre-trained target detection algorithm may adopt target detection algorithms such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), RetinaNet, DETR (DEtectionTRansformer), etc.;

[0062] For example, the YOLO target detection algorithm is used, which is an algorithm that converts images Split into a grid and predict multiple bounding boxes and their class probabilities for each grid cell;

[0063] Its loss function is as follows:

[0064] ;

[0065] in, is the size of the grid after segmentation, is the number of bounding boxes predicted per grid cell; and They are the indicator functions of whether the object exists or not, , , , are the true bounding box coordinates and dimensions, , , , is the corresponding predicted value, is the object confidence, is the corresponding predicted value, and It is a hyperparameter that adjusts the weights of different loss terms;

[0066] S300, intelligent sorting mechanism:

[0067] For each object in each image , according to its standardized characteristics and user text description Keyword matching in , combined with multimodal impact factors , calculate a comprehensive score ;

[0068] Comprehensive scoring formula: ;

[0069] in It is the average feature vector of all objects, which is used to measure the similarity of object features relative to the global feature space; is a similarity function between the object features and the average features, such as cosine similarity or Euclidean distance; , , They are the weights of feature similarity, keyword matching, and multimodal influence; is the average feature vector of all objects;

[0070] According to the comprehensive rating Sort the objects in all images and generate a sorted list of objects ;

[0071] S400, first batch generation of image groups:

[0072] Use personalized generative models for specific genres or domains ;

[0073] Input sorted list of objects and its priority, as well as multimodal information as auxiliary input to generate a set of new images that meet the requirements for each original image;

[0074] Personalized Generative Model The input is represented as ,in is a sorted list of objects, It is audio information. It is video information;

[0075] Output a set of newly generated pictures ;

[0076] In one embodiment of the present invention, the personalized generation model It is a conditional generative adversarial network (cGAN), which is a personalized generative model Includes the following:

[0077] 1. Personalized Generative Model The structure:

[0078] Generator : The random noise vector and condition information As input, the output generated image is ;

[0079] Discriminator : Receive real or generated images and condition information , output a probability value , indicating the given condition Next picture is the probability of being true;

[0080] 2. Loss function:

[0081] Generator Loss : The goal is to make the generator generate pictures that are as realistic as possible to deceive the discriminator;

[0082] Discriminator loss : The purpose is to allow the discriminator to accurately distinguish between real pictures and generated pictures;

[0083] For the conditional generative adversarial network, the following loss function is defined:

[0084] ;

[0085] ;

[0086] ;

[0087] in: Represents random noise The distribution of is usually the standard normal distribution; Represents intermediate data; Indicates conditional information The distribution of , which can be text descriptions, category labels, or other forms of information; Represents the real data distribution;

[0088] 3. Parameter update:

[0089] Generator Update: Minimization , that is, maximize:

[0090] ;

[0091] Discriminator update: Minimize , that is, maximizing simultaneously:

[0092] ;

[0093] .

[0094] in represents a random noise vector, which comes from a standard normal distribution ; Represents conditional information, including embedded vectors of text descriptions, classification labels, audio features, video frame features, etc. Represents a generated image or a real image.

[0095] In one embodiment of the present invention, the personalized generation model The training process is as follows:

[0096] Alternating Update Generator and the discriminator : In each iteration, first fix the generator parameters , update the discriminator parameters To minimize ; Then fix the discriminator parameters , update the generator parameters To minimize ;

[0097] Monitoring indicators: Regularly check the quality of generated images and record the trend of training loss;

[0098] The authenticity and diversity of generated images can be evaluated using indicators such as FID (Fréchet Inception Distance) or IS (Inception Score) on the validation set;

[0099] Save checkpoints: Save model parameters periodically during training so that you can resume training or test later.

[0100] In the personalized generative model Condition information May include multimodal information provided by the user (such as text description , Audio ,video ), and object features extracted from the original image , so the input of the generator is expressed as:

[0101] ;

[0102] in Represents the vector after all conditional information is fused. This fusion can be achieved in many ways, such as through attention mechanism, concatenation operation or learning a mapping function to convert information of different modalities into a unified feature space.

[0103] S500, secondary selection and set formation:

[0104] The image collection generated after processing , select a batch of pictures that are closest to the expected effect to form a new candidate set ;

[0105] The secondary selection process is to calculate the similarity score To achieve this, for the jth processed image , calculate its difference with the user text description The matching degree of the two modalities is considered while considering the influence of multimodal information;

[0106] Similarity score formula: ;

[0107] in Generate pictures The data characteristics, is a function to measure similarity;

[0108] In this step, the object features extracted previously are used To evaluate the similarity between the generated image and the original object to ensure the consistency of the generated content;

[0109] Output a series of candidate sets ;

[0110] S600, feature analysis and parameter adjustment:

[0111] For the i-th candidate set Conduct in-depth feature analysis to determine what content meets user expectations and what content needs to be improved, and update the personalized generation model Parameters of to reflect the new set of feature parameters, including how to better combine multimodal information and feature similarity;

[0112] Integrate user expectations into text descriptions Updated to , if there is additional modal information, update the condition information For new condition information ;

[0113] S700, final destination image batch generation:

[0114] Use the updated personalized generative model Run again, output the final destination image for each original image, and output the final generated image set ;

[0115] S800, Evaluation and Feedback Loop:

[0116] 1. A collection of pictures will be generated Show it to users and get feedback;

[0117] 2. If the user feedback is satisfied, the generated image is the target image; if the user feedback is dissatisfied, the customer feedback content is integrated into the text description Then, perform steps S100-S800.

[0118] like Figure 2 and Figure 3 As shown, in one embodiment of the present invention, an AI-based intelligent image processing system is also proposed, including:

[0119] User input and data preparation module 101: collects and pre-processes user input data;

[0120] Batch object detection and classification module 102: identifies and classifies objects in all original images, and extracts features;

[0121] Intelligent sorting module 103: sorting objects according to object features, text matching and multimodal influence;

[0122] Initial generation module 104: generating a set of new images for each original image based on the sorted object list and multimodal information;

[0123] Secondary selection and candidate set forming module 105: selecting pictures closest to the expected effect from the pictures generated initially to form a candidate set;

[0124] Feature analysis and parameter adjustment module 106: performs in-depth analysis on the candidate set to determine the direction of improvement and adjust the parameters of the generated model;

[0125] Final generation module 107: using the updated model, outputs a final destination image for each original image;

[0126] Evaluation and feedback loop module 108: Display the generated images to the user, obtain feedback, and further optimize the results as needed.

[0127] In one embodiment of the present invention, the following example of applying the above processing method and processing system is given, the purpose of which is to develop an AI-based image personalization generation model for interior designers to generate furniture layout plans that meet a specific style based on room photos and descriptions provided by users;

[0128] 1. User input and data preparation module:

[0129] User-uploaded images: Interior designers upload a set of renderings and photos of existing rooms similar to the target image , including photos Showing the room layout from different angles;

[0130] Multimodal information collection:

[0131] Text description : "A modern minimalist living room with a large window, a grey sofa and a wooden coffee table."

[0132] Audio Information : Background music (such as relaxing piano music) to convey the atmosphere.

[0133] Video Information : A short video showing the designer’s preferred furniture placement.

[0134] Data preprocessing unit: standardize all images (such as resizing and normalization) and extract features from audio and video (such as emotion, rhythm, object movement trajectory, etc.).

[0135] 2. Batch object detection and classification module:

[0136] Object Detection Engine: Identifies objects in each image, such as walls, windows, doors, existing furniture, etc.

[0137] Classifier: classifies the identified objects into different categories (such as window, sofa, table, etc.);

[0138] Feature Extractor: Extracts standardized features for each object , including color distribution, texture, shape, size, position, etc.;

[0139] 3. Intelligent sorting module:

[0140] Rating calculator: Based on object characteristics and user text description Keyword matching in , combined with multimodal impact factors , calculate the comprehensive score for each object .

[0141] Sorting engine: sort all objects according to their comprehensive scores and generate a sorted list of objects For example, large windows, a gray sofa, and a wooden coffee table might be ranked first.

[0142] 4. Generate module for the first time:

[0143] Personalized Generative Model : Accepts a sorted list of objects and its priority, combining multimodal information and , output a set of newly generated pictures .

[0144] The images show different styles of furniture arrangement, but all highlight the key elements mentioned by users (such as large windows, gray sofas and wooden coffee tables).

[0145] 5. Secondary selection and candidate set formation module:

[0146] Similarity Evaluator: Calculates the similarity score for each generated image , the generated image is evaluated for its similarity to the original object and its match to the user description.

[0147] Selector: Select the best matching image to form a candidate set .

[0148] For example, select images that best represent a "modern minimalist style" with a specific piece of furniture placed in a reasonable manner.

[0149] 6. Feature analysis and parameter adjustment module:

[0150] Analysis tool: Analyze the content in the candidate set that meets the user's expectations and the content that needs to be improved. For example, it is found that although some generated images have gray sofas, the overall color tone is not harmonious enough.

[0151] Parameter Optimizer: Update Personalized Generative Model to reflect the new set of feature parameters. For example, increase the weight of color consistency.

[0152] 7. Final generation module:

[0153] Generator: Run the personalized generative model again , considering the influence of multimodal information, output the final generated picture set .

[0154] These images not only match the user's description, but are also more visually harmonious and unified.

[0155] 8. Evaluation and feedback loop module:

[0156] Display interface: Display the final generated image through the user's acquisition interface For interior designers.

[0157] Feedback Collector: Get feedback from designers to understand the actual effect of the generated results. For example, the designer may point out that a certain detail needs to be adjusted.

[0158] Iterative optimizer: Fine-tune the model structure or hyperparameters based on the evaluation results and retrain the model if necessary to further improve the generation quality.

[0159] Results: After the above steps, the system generates a series of high-quality furniture arrangement plan images for interior designers. Each plan reflects a modern and simple style and accurately contains key elements that users want, such as large windows, gray sofas, and wooden coffee tables. Designers can make further designs and adjustments based on these plans, thereby greatly saving design time and costs.

[0160] At least one embodiment of the present disclosure provides a storage medium storing non-transitory computer-readable instructions for executing one or more steps in the aforementioned AI-based intelligent image processing method.

[0161] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

[0162] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are within the protection of the present embodiment.

Claims

1. An AI-based intelligent image processing method, characterized in that: The following steps are involved: S100, preparing an original picture set and multimodal description: a user provides input data, the input data includes original pictures, text descriptions and multimodal information; S200, Batch Object Detection, Classification and Feature Extraction: Use pre-trained object detection algorithms to batch process object recognition in all original images; classify the objects recognized in each image and label them as belonging to the same category; extract a set of standardized features for each type of object; S300, intelligent sorting mechanism: for each object in each picture, a comprehensive score is obtained based on its standardized features and the keyword matching degree in the user's text description, combined with multimodal influence factors; Sort the objects in all images according to their comprehensive scores and generate a sorted object list; S400, initial batch generation of image groups: using a personalized generation model for a specific type or field, inputting a sorted list of objects and their priorities, and multimodal information as auxiliary input, to generate a set of new images that meet the requirements for each original image; S500, secondary selection and set formation: for the new images that meet the requirements after processing, secondary selection is performed by calculating the similarity score, and a group of images that are closest to the expected effect are selected to form a new candidate set; S600, feature analysis and parameter adjustment: in-depth feature analysis of the candidate set, determine the content that meets the user's expectations and the content that needs to be improved, and update the parameters in the personalized generation model; S700, batch generation of final destination images: using the updated personalized generation model to run again, outputting the final destination image for each original image, and outputting the final generated image set; S800, evaluation and feedback loop: display the generated picture set to the user. If the user feedback is satisfactory, the generated picture is the target picture. If the user feedback is dissatisfied, the customer feedback content is integrated into the text description, and then returns to step S100.

2. The AI-based intelligent image processing method according to claim 1, characterized in that: Multimodal information includes audio and video.

3. The AI-based intelligent image processing method according to claim 2, characterized in that: Standardized features include color distribution, texture, shape, size, and position.

4. The AI-based intelligent image processing method according to claim 3, characterized in that: The personalized generative model includes a generator and the discriminator , generator is the random noise vector and condition information As input, the output generated image is ; Discriminator Is to receive real or generated images and condition information , output a probability value , indicating the given condition Next picture is the true probability.

5. The AI-based intelligent image processing method according to claim 4, characterized in that: For the conditional generative adversarial network, the following loss function is defined: ； ； ； in Represents random noise The distribution of is usually the standard normal distribution; Represents intermediate data; Indicates conditional information The conditional information includes text description, category label or other forms of information; Represents the actual data distribution.

6. The AI-based intelligent image processing method according to claim 5, characterized in that: The personalized generation model also requires parameter updates: including generator updates and discriminator updates; Generator updates are minimizations of ,maximize: ； The discriminator update is to minimize , while maximizing: ； ； in Represents a random noise vector, which comes from a standard normal distribution ; Representation of conditional information, including embedded vectors of text descriptions, classification labels, audio features, and video frame features; Represents a generated image or a real image.

7. The AI-based intelligent image processing method according to claim 6, characterized in that: The similarity score formula in the secondary selection is as follows: ； in Generate pictures The data characteristics, is a function to measure similarity, is the jth image generated after processing.

8. The AI-based intelligent image processing method according to claim 7, characterized in that: Integrate user expectations into text descriptions Updated to , if there is additional modal information, update the condition information For new condition information .

9. An AI-based intelligent image processing system, used to execute an AI-based intelligent image processing method as claimed in any one of claims 1 to 8, characterized in that: include: User input and data preparation module: collect and preprocess user input data; Batch object detection and classification module: identifies, classifies, and extracts features of objects in all original images; Intelligent sorting module: sort objects according to object features, text matching and multimodal influence; Initial generation module: Generate a set of new images for each original image based on the sorted object list and multimodal information; Secondary selection and candidate set formation module: select the pictures closest to the expected effect from the initially generated pictures to form a candidate set; Feature analysis and parameter adjustment module: conducts in-depth analysis of the candidate set to determine the direction of improvement and adjust the parameters of the generated model; Final generation module: Use the updated model to output the final destination image for each original image; Evaluation and feedback loop module: Show the generated images to users, get feedback, and further optimize the results as needed.

10. A storage medium storing non-transitory computer-readable instructions for executing an AI-based intelligent image processing method as described in any one of claims 1-8.

Images