Image processing method and device, equipment, storage medium and chip

Abstract

The invention relates to an image processing method and device, equipment, a storage medium and a chip, and the method comprises the steps: obtaining a to-be-processed image which comprises m first objects, and m is a positive integer; a pre-classification model is called to pre-classify the m first objects, pre-classification results corresponding to the m first objects are obtained, and the pre-classification results are used for indicating the types corresponding to the first objects; and based on the pre-classification results corresponding to the m first objects, performing image processing on a second object in the to-be-processed image to obtain a processing result corresponding to the second object, the second object being an object in the m first objects. Therefore, the technical problems of low image processing accuracy and the like in related technologies can be solved.

Description

Image processing methods, apparatus, devices, storage media and chips Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to an image processing method, apparatus, device, storage medium, and chip. Background Technology

[0002] Computer vision is an integral part of various intelligent / autonomous systems in fields such as manufacturing, document analysis, and medical diagnosis. It is the study of using cameras / video cameras and computer equipment to acquire the data and information we need from the objects being photographed. Figuratively speaking, it's about equipping computers with eyes (cameras / video cameras) and a brain (algorithms) to replace human eyes in recognizing, tracking, and measuring target objects, thereby enabling computers to perceive their environment.

[0003] When computer vision is applied to image processing, such as image recognition, question answering, and search, it usually involves directly processing the input image. Taking image recognition as an example, it mainly classifies based on the image features of the input image. However, since image features are easily affected by factors such as background noise, the accuracy of image processing is not high. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, this disclosure provides an image processing method, apparatus, device, storage medium, and chip to solve the technical problems of low image processing accuracy in the aforementioned related technologies.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring an image to be processed, the image to be processed including m first objects, where m is a positive integer; calling a pre-classification model to pre-classify the m first objects to obtain pre-classification results corresponding to each of the m first objects, the pre-classification results indicating the type to which the corresponding first object belongs; and performing image processing on a second object in the image to be processed based on the pre-classification results corresponding to each of the m first objects to obtain a processing result corresponding to the second object, wherein the second object is an object among the m first objects. Optionally, the image processing includes at least one of classification recognition, question answering processing, translation, text extraction, interactive search, and guided search.

[0006] In some embodiments, the step of performing image processing on the second object in the image to be processed based on the pre-classification results corresponding to each of the m first objects to obtain the processing result corresponding to the second object includes: determining the priority of the second object in the image to be processed; and performing image processing on the second object based on the pre-classification result of the second object and the priority of the second object to obtain the processing result.

[0007] In some embodiments, determining the priority of the second object in the image to be processed includes: after receiving an operation instruction for any first object, identifying any first object as the second object; and determining the priority of the second object as a preset priority.

[0008] In some embodiments, determining the priority of the second object in the image to be processed includes: calculating the proportion of each of the m first objects in the image to be processed to obtain the proportion of each of the m first objects; determining the second object and the priority of the second object based on the proportion of each of the m first objects; wherein the priority of the second object and the proportion of the second object are positively correlated.

[0009] In some embodiments, determining the priority of the second object in the image to be processed includes: obtaining user question information; performing intent understanding and priority determination on m first objects in the image to be processed based on the user question information to obtain the second object and the priority of the second object.

[0010] In some embodiments, the method further includes: acquiring a training set, the training set including at least n sample images, where n is a positive integer; pre-training and aligning the n sample images in the training set based on the pre-classification model to be trained, to obtain alignment features corresponding to each of the n sample images; performing supervised fine-tuning on the alignment features corresponding to each of the n sample images, to obtain fine-tuned features corresponding to each of the n sample images; and performing preference optimization on the fine-tuned features corresponding to each of the n sample images to adjust the model parameters of the pre-classification model to be trained, thereby obtaining the trained pre-classification model.

[0011] In some embodiments, the pre-training alignment of the n sample images in the training set based on the pre-classification model to be trained to obtain the alignment features corresponding to each of the n sample images includes: encoding the image features of the n sample images in the training set based on the pre-classification model to be trained to obtain the image features of each of the n sample images in the image space; aligning and spatially transforming the image features of each of the n sample images in the image space to obtain the alignment features corresponding to each of the n sample images; wherein, the alignment features are the image features corresponding to the sample images in the text space.

[0012] In some embodiments, the training set further includes historical question information and historical question answers for the sample images, wherein the historical question answers include positive sample answers and negative sample answers, and the preference optimization of the fine-tuning features corresponding to each of the n sample images includes: performing positive sample enhancement and negative sample reduction processing on the fine-tuning features corresponding to each of the n sample images based on the historical question information, positive sample answers, and negative sample answers for each of the n sample images.

[0013] In some embodiments, the method further includes: obtaining follow-up information of the image to be processed; and searching for answers to the follow-up information based on the pre-classification results corresponding to each of the m first objects in the image to be processed and / or the processing results corresponding to the second objects, to obtain corresponding follow-up answers.

[0014] According to a second aspect of the present disclosure, an image processing method is provided, which can be applied to a terminal device. The method includes: after receiving a wake-up command from a voice assistant, displaying an interactive interface, the interactive interface including a target control, the wake-up command being used to wake up the voice assistant of the terminal device, the target control being a functional control for acquiring an image; after receiving an operation command for the target control, acquiring an image to be processed, and performing corresponding pre-classification and image processing based on m first objects included in the image to be processed, where m is a positive integer.

[0015] In some embodiments, the method further includes: receiving an operation instruction for any of the first objects in the image to be processed; and, in response to the operation instruction for any of the first objects, determining any of the first objects as a second object to perform image processing on the second object.

[0016] According to a third aspect of the present disclosure, an image processing apparatus is provided, comprising: an acquisition module configured to acquire an image to be processed, the image to be processed including m first objects, where m is a positive integer; a processing module configured to invoke a pre-classification model to pre-classify the m first objects, obtaining pre-classification results corresponding to each of the m first objects, the pre-classification results indicating the type to which the corresponding first object belongs; the processing module further configured to perform image processing on a second object in the image to be processed based on the pre-classification results corresponding to each of the m first objects, obtaining a processing result corresponding to the second object, the second object being an object among the m first objects. Optionally, the image processing includes at least one of classification recognition, question answering processing, translation, text extraction, interactive search, and guided search.

[0017] For any content not introduced or described in the embodiments of this disclosure, please refer to the relevant descriptions in the foregoing method embodiments. This disclosure does not limit the scope of the embodiments.

[0018] According to a fourth aspect of the present disclosure, an image processing apparatus is provided, comprising: a processing module configured to display an interactive interface after receiving a wake-up command from a voice assistant, the interactive interface including a target control, the wake-up command being used to wake up the voice assistant of a terminal device, the target control being a functional control for acquiring an image; and an acquisition module configured to acquire an image to be processed after receiving an operation command for the target control, and to perform corresponding pre-classification and image processing based on m first objects included in the image to be processed, where m is a positive integer.

[0019] For any content not introduced or described in the embodiments of this disclosure, please refer to the relevant descriptions in the foregoing method embodiments. This disclosure does not limit the scope of the embodiments.

[0020] According to a fifth aspect of the present disclosure, a computer device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the steps of the image processing method provided in the first or second aspect described above.

[0021] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the image processing method provided in the first or second aspect of the present disclosure.

[0022] According to a seventh aspect of the present disclosure, a chip is provided, comprising: a processor and an interface; the processor is configured to read instructions to execute the steps of the image processing method provided in the first or second aspect described above.

[0023] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: a computer device acquires an image to be processed, the image to be processed including m first objects, where m is a positive integer; a pre-classification model is invoked to pre-classify the m first objects, obtaining pre-classification results corresponding to each of the m first objects, the pre-classification results indicating the type to which the corresponding first object belongs; based on the pre-classification results corresponding to each of the m first objects, image processing is performed on a second object in the image to be processed, obtaining a processing result corresponding to the second object, where the second object is an object among the m first objects. Thus, this disclosure allows for pre-classification of objects in the image to be processed before image processing, thereby improving the effect and accuracy of image processing.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] Figure 1 is a schematic diagram of a scenario operation according to an exemplary embodiment.

[0027] Figure 2 is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0028] Figure 3 is a schematic diagram of the internal structure of a pre-classification model according to an exemplary embodiment.

[0029] Figure 4 is a schematic diagram of an image processing flow according to an exemplary embodiment.

[0030] Figure 5 is a schematic diagram of a priority determination process according to an exemplary embodiment.

[0031] Figure 6 is a schematic diagram of an interface for a user to select a second object according to an exemplary embodiment.

[0032] Figure 7 is a schematic diagram illustrating another priority determination process according to an exemplary embodiment.

[0033] Figure 8 is a schematic diagram illustrating another priority determination process according to an exemplary embodiment.

[0034] Figure 9 is a flowchart illustrating a follow-up question answering process according to an exemplary embodiment.

[0035] Figure 10 is a flowchart illustrating a pre-classification model training method according to an exemplary embodiment.

[0036] Figure 11 is a schematic diagram of the internal process of model training according to an exemplary embodiment.

[0037] Figure 12 is a flowchart illustrating another image processing method according to an exemplary embodiment.

[0038] Figure 13 is a schematic diagram illustrating the acquisition of an image to be processed according to an exemplary embodiment.

[0039] Figure 14 is a schematic diagram of the structure of an image processing apparatus according to an exemplary embodiment.

[0040] Figure 15 is a schematic diagram of the structure of another image processing apparatus according to an exemplary embodiment.

[0041] Figure 16 is a schematic diagram of the structure of another image processing apparatus according to an exemplary embodiment.

[0042] Figure 17 is a schematic diagram of the structure of a chip according to an exemplary embodiment. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0044] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are performed with the authorization of the respective device owner.

[0045] Currently, some terminal devices have general visual intelligence functions such as image recognition, question answering, and search. However, each function requires calling the corresponding model page separately. For example, question answering calls ChatGPT separately, and search calls the Google search interface separately, resulting in a cluttered user interface.

[0046] To address the aforementioned problems, this disclosure provides an image processing method, apparatus, device, storage medium, and chip. Some embodiments applicable to this disclosure are described below.

[0047] Image processing refers to the techniques used in computer equipment to analyze, process, and improve images to achieve desired results. Image processing can include, but is not limited to, any one or a combination of the following: classification and recognition (also known as image recognition), question answering (also known as image question answering), translation, text extraction, interactive search, guided search, or other image processing methods.

[0048] The aforementioned classification and recognition refers to the technology of using computer equipment to process, analyze, and understand images in order to identify objects contained within them. This classification and recognition primarily relies on computer vision and artificial intelligence algorithms, which automatically identify and classify objects in images by learning their features. This disclosure does not limit the specific implementation methods of the aforementioned classification and recognition; for example, feature extraction algorithms, deep learning algorithms, or other custom algorithms for image classification and recognition can be used, and this disclosure will not impose further limitations or details.

[0049] The aforementioned question-answering processing refers to the ability to understand and answer user questions based on computer vision technology. This technology can extract information from large amounts of data, analyze and understand the user's intent in asking the question, and generate precise and concise answers, thereby providing efficient and accurate responses. This disclosure does not limit the specific implementation methods of the aforementioned question-answering processing, such as obtaining results through training and inference using corresponding models and historical question-answering data, etc., and this disclosure does not impose excessive limitations in this regard.

[0050] The aforementioned translation can also refer to image translation, specifically the process of converting an image from one representation to another; that is, the transformation from one image to another, such as converting image style, color, and other visual features. This disclosure does not limit the specific implementation of the aforementioned translation; for example, a preset image conversion algorithm can be used for conversion processing, such as using a grayscale processing algorithm to convert a color image to a grayscale image, etc. This disclosure does not impose further limitations or details in this regard.

[0051] The aforementioned text extraction refers to converting text information in an image into an editable text format using computer vision and pattern recognition technologies. This disclosure does not limit the specific implementation methods of the aforementioned text extraction; for example, optical character recognition (OCR) technology or other algorithms or technologies used for text extraction may be employed, and this disclosure will not impose further limitations or details in this regard.

[0052] The aforementioned interactive search can refer to a visual search, where users interact with computer devices to search for recommendations, such as finding products they need or searching for answers. This disclosure does not limit the specific implementation of the aforementioned interactive search. For example, after detecting that a user interacts with any object on the computer device's interface (e.g., selecting, clicking, etc.), recommended products matching that object can be searched. This disclosure does not impose further limitations or provide detailed descriptions in this regard.

[0053] The aforementioned guided search can refer to searching for specific keywords through user operation. This disclosure does not limit the specific implementation of the aforementioned guided search. For example, it can also use user-selected objects / keywords to guide the search for matching recommendation information, such as searching for and recommending songs based on selected keywords, etc. This disclosure will not impose further limitations or details on this.

[0054] The interactive search and guided search mentioned above are both types of image search, and the image search can also include other custom search methods. The image search mentioned above can refer to a technology that provides users with retrieval services for relevant graphic image materials on the Internet by searching for image text or visual features. This disclosure does not impose too many limitations on the specific implementation methods of the image search mentioned above; for example, deep learning algorithms can be used to implement the search, etc., and this disclosure does not impose too many limitations or details in this regard.

[0055] The following is an exemplary schematic diagram of a scenario to which this disclosure applies. Taking the above image processing, including image recognition and question-answering processing, as an example, please refer to Figure 1, which is a schematic diagram of a scenario operation according to an exemplary embodiment. As shown in Figure 1, the terminal device can provide a designated icon (illustrated as a camera icon) as an entry point for observing the external world. After the user clicks on the designated image (camera icon) and takes a picture, the terminal device can upload the image captured by the user to a cloud server. The cloud server can classify and recognize the image captured by the user using a pre-classification method, and optionally can also perform image processing such as image translation, text extraction, interactive search, and guided search to obtain the corresponding processing results. The cloud server returns the processing results to the terminal device for display. Specific implementation methods for image processing will be described in detail below in this disclosure, and will not be repeated here. It can be seen that compared with the interface operation of the prior art, this disclosure can simplify the interactive interface, reduce user operations, and thus improve the user experience.

[0056] Please refer to Figure 2, which is a flowchart illustrating an image processing method according to an exemplary embodiment. The method shown in Figure 2 can be applied to a computer device and may include the following steps: S201, acquiring an image to be processed, wherein the image to be processed includes m first objects, where m is a positive integer.

[0057] The image to be processed as described in this disclosure can refer to any image in a computer device that needs to undergo image processing, and the image includes one or more (first) objects. This disclosure does not limit the number of the first objects; it can be determined according to the actual situation, specifically one or more, etc. The following description uses an image to be processed containing m first objects as an example, which does not constitute a limitation. m is a positive integer that the system or user defines according to the actual situation. For example, it can be an empirical value set based on user experience, or a statistical value calculated based on a series of experimental data, etc. This disclosure does not impose further limitations or details on this.

[0058] The objects involved in this disclosure may refer to targets or objects included in the image to be processed, which may include, but are not limited to, animals, plants, people, vehicles, streetlights, or other custom objects.

[0059] This disclosure does not limit the implementation method of obtaining the above-mentioned image to be processed. For example, a computer device can receive the image to be processed from other devices (such as terminal devices or servers) through a network, or the image to be processed can be obtained directly from a local database.

[0060] The computer equipment involved in this disclosure may refer to terminal equipment, server equipment, or other equipment with image processing capabilities; this disclosure does not impose any limitations on this. The aforementioned server equipment may include, but is not limited to, local servers, cloud servers, or servers deployed in other forms. The aforementioned terminal equipment may also be referred to as terminal equipment, user equipment (UE), mobile station (MS), mobile terminal equipment (MT), etc. Terminal equipment may include automobiles with communication functions, intelligent vehicles, mobile phones, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal equipment, augmented reality (AR) terminal equipment, wireless terminal equipment in industrial control, wireless terminal equipment in self-driving, wireless terminal equipment in remote medical surgery, wireless terminal equipment in smart grids, wireless terminal equipment in transportation safety, wireless terminal equipment in smart cities, wireless terminal equipment in smart homes, and so on. The embodiments disclosed herein do not limit the specific technology or form of the computer device.

[0061] S202. Call the pre-classification model to pre-classify m of the first objects to obtain the pre-classification results corresponding to each of the m first objects. The pre-classification results are used to indicate the type to which the corresponding first object belongs.

[0062] In this disclosure, the aforementioned pre-classification model can refer to a model used for pre-classifying objects in an image. This model may include, but is not limited to, models such as the K-nearest neighbor algorithm, convolutional neural network, backpropagation neural network, or other custom models for image pre-classification. This disclosure does not impose further limitations or details on these models. The pre-classification results can specifically indicate the classification type or category to which the corresponding first object belongs. This disclosure does not limit the type / category of the corresponding classification; it can be pre-configured by the system, such as categorizing objects, schedules, topics, or other types. This disclosure does not impose further limitations or details on these types.

[0063] S203. Based on the pre-classification results corresponding to each of the m first objects, perform image processing on the second object in the image to be processed to obtain the processing result corresponding to the second object, where the second object is one of the m first objects. Optionally, the image processing may include at least one of classification recognition, question answering, translation, text extraction, interactive search, and guided search.

[0064] In this disclosure, the second object can refer to any object in the image to be processed that requires image processing; that is, the second object can be any one or more of the m first objects in the image to be processed. This disclosure does not limit the number of the second objects; it can be determined according to the actual situation. For ease of description, this disclosure uses one second object as an example to illustrate the relevant content, but this does not constitute a limitation.

[0065] The aforementioned processing result refers to the result obtained by performing corresponding image processing on the second image. Taking the image processing including classification and recognition as an example, in this case, the processing result is used to indicate the specific category to which the second object belongs, such as the second object being a person, animal, plant, or vehicle, etc. As another example, taking the aforementioned image processing including question-and-answer processing as an example, in this case, the processing result is used to indicate the response answer output by performing question-and-answer processing on the second object; for example, if the second object is a car and a user asks what kind of car it is, after processing, the processing result can return the corresponding response answer, such as XX brand car, etc. For a description of image processing, please refer to the relevant descriptions in the foregoing embodiments of this disclosure, which will not be repeated here.

[0066] By implementing embodiments of this disclosure, a computer device acquires an image to be processed, the image to be processed including m first objects, where m is a positive integer; a pre-classification model is invoked to pre-classify the m first objects, obtaining pre-classification results corresponding to each of the m first objects, the pre-classification results indicating the type to which the corresponding first object belongs; based on the pre-classification results corresponding to each of the m first objects, image processing is performed on a second object in the image to be processed, obtaining a processing result corresponding to the second object, where the second object is an object among the m first objects. Thus, this disclosure allows for pre-classification of objects in the image to be processed before image processing, thereby improving the effect and accuracy of image processing.

[0067] In step S202, this disclosure does not limit the internal structure of the pre-classification model. For example, please refer to Figure 3, which is a schematic diagram of the internal structure of a pre-classification model according to an exemplary embodiment. The pre-classification model 300 shown in Figure 3 may include an encoder 301 (Vision encoder), an adapter 302, a Large Language Model (LLM) 303, and a decoder 304 (Vision decoder). The encoder 301 is mainly used to encode the input image into image features in the image space. The adapter 302 is mainly used to convert or map the image features in the image space into image features in the text space. The Large Language Model 303 is mainly used to perform processing such as feature extraction and classification on the image features in the text space, thereby obtaining processed features in the text space. The decoder 304 is mainly used to decode the processed features in the text space into processed features in the image space and classify them to obtain the corresponding pre-classification result.

[0068] This disclosure does not impose excessive limitations or details on the functions and internal structures of the components in the aforementioned pre-classification model. For example, the encoder may include a transformer architecture, the adapter may include a two-layer fully connected neural network, and the decoder may include an attention mechanism architecture, etc. This disclosure does not impose excessive limitations or details on these aspects.

[0069] In step S203, this disclosure does not limit the specific implementation of the above image processing. For example, image processing such as image classification and recognition can be directly performed on the second object based on the pre-classification result of the second object to obtain the corresponding processing result. For example, please refer to Figure 4, which is a schematic flowchart of an image processing according to an exemplary embodiment. The flowchart shown in Figure 4 may include the following implementation steps: S401, determining the priority of the second object in the image to be processed.

[0070] This disclosure does not limit the order in which the second object and its priority are determined. For example, the second object that needs to be processed in the image to be processed can be determined first, and then the priority of the second object can be determined. Or, the second object and its priority can be determined at the same time. This disclosure does not limit or elaborate on these matters.

[0071] This disclosure does not limit the implementation methods for determining the second object and the priority of the second object as described above. Several possible implementation methods are described below by example.

[0072] In one implementation, please refer to FIG5, which is a schematic diagram of a priority determination process according to an exemplary embodiment. The process shown in FIG5 may include the following implementation steps: S501, after receiving an operation instruction for any of the first objects, determine any of the first objects as the second objects.

[0073] Upon receiving / detecting an operation instruction for any of the aforementioned first objects, this disclosure can determine the aforementioned first objects as the aforementioned second objects requiring further image processing. The aforementioned operation instruction may refer to an instruction generated after a user performs a specified operation (e.g., selection, movement, single click, double click, etc.) on any of the first objects in the image to be processed.

[0074] S502, Set the priority of the second object to a preset priority.

[0075] After identifying the second object, this disclosure can determine / configure the priority of the second object to a preset priority, such as the highest priority supported by the system. The preset priority can refer to a priority pre-defined by the system or the user, such as the highest priority supported by the system, such as level 10.

[0076] For example, please refer to Figure 6, which is a schematic diagram of an interface for a user to select a second object according to an exemplary embodiment. As shown in Figure 6, after the user selects a girl in the image to be processed as the second object, the girl can be directly assigned the highest priority, and subsequent image processing, such as classification and recognition, can be performed on the girl, the second object with the highest priority. Other objects in the image to be processed (such as a boy) can be treated as lower priority objects to be processed, or they can be directly ignored, etc., which will not be limited or described in detail in this disclosure.

[0077] In another embodiment, please refer to FIG7, which is a schematic diagram of another priority determination process according to an exemplary embodiment. The process shown in FIG7 may include the following implementation steps: S701, calculate the proportion of m first objects in the image to be processed to obtain the proportion corresponding to each of the m first objects.

[0078] This disclosure can calculate the proportion of m first objects in the image to be processed, obtaining the proportion of each of the m first objects in the image to be processed. This disclosure does not limit the specific implementation of the proportion calculation. For example, this disclosure can use an object detection algorithm to identify and calculate the area occupied by each of the m first objects in the image to be processed, and then calculate the proportion of each of the m first objects by comparing it with the total area of ​​the image to be processed. The object detection algorithm may include, but is not limited to, region convolutional neural networks, YOLO detection, or other algorithms used for object detection, etc., and this disclosure will not limit or elaborate on them further.

[0079] S702. Based on the proportions of each of the m first objects, determine the second object and its priority; wherein the priority of the second object is positively correlated with its proportion.

[0080] This disclosure does not limit the implementation of the determination of the second object and the priority of the second object. In practical applications, the priority of the second object is positively correlated with the proportion of the second object. Generally, the larger the proportion of the second object, the higher the priority of the second object; conversely, the smaller the proportion of the second object, the lower the priority of the second object.

[0081] For example, when the second object includes m of the first objects, this disclosure can sort the proportions of the m first objects in descending order, and use the sorting results as the priority of the respective objects. Alternatively, the second object is usually the object with the largest proportion among the m first objects, that is, the largest first object in the image to be processed; and the priority of this second object is directly determined / configured as a preset priority, such as the highest priority supported by the system, to facilitate subsequent image processing of the second image.

[0082] In another implementation, please refer to FIG8, which is a schematic diagram of another priority determination process according to an exemplary embodiment. The process shown in FIG8 may include the following implementation steps: S801, obtaining user problem information.

[0083] In this disclosure, the aforementioned user question information can refer to the question information that the user wants to ask, such as what the second object is, the location of the second object in the image to be processed, etc. This disclosure does not limit the implementation method for obtaining the aforementioned user question information. For example, it can be directly input by the user, obtained from other devices via a network, or it can be question information set / stored by default in the multimodal model, etc. This disclosure does not impose further limitations or details on these aspects. The introduction to the multimodal model will be detailed below and will not be described here.

[0084] S802. Based on the user question information, perform intent understanding and priority determination on m first objects in the image to be processed to obtain the second object and the priority of the second object.

[0085] This disclosure can combine the aforementioned user question information to perform user intent understanding and analysis on the m first objects in the aforementioned image to be processed, thereby obtaining the aforementioned second objects; and then determine / configure the priority of the aforementioned second objects as a preset priority, such as the highest priority supported by the system.

[0086] This disclosure does not limit the specific implementation of the above-mentioned intent understanding. For example, referring expression comprehension (REC) can be used in combination with user question information to understand and analyze the user's true intent in the image to be processed, thereby determining the second object that needs to be processed. This disclosure does not limit or elaborate on this.

[0087] S402. Based on the pre-classification result of the second object and the priority of the second object, perform image processing on the second object to obtain the processing result.

[0088] This disclosure does not limit the specific implementation of step S402. For example, based on the pre-classification result of the second object, this disclosure can dynamically call the corresponding multimodal model to perform further image processing on the second object in the image to be processed according to the priority of the second object. The multimodal model and the pre-classification result (specifically, the classification type indicated by the pre-classification result) are in one-to-one correspondence, and different multimodal models can be dynamically designed / connected for different classification types.

[0089] For example, if the pre-classification result includes object types such as animals, plants, people, and vehicles, the multimodal model can be an object multimodal model. If the pre-classification result is a schedule type, the multimodal model can be a schedule recognition multimodal model. If the pre-classification result is a question type, the multimodal model can be a question-solving multimodal model. If the pre-classification result is other types, the multimodal model can be other multimodal models, and so on. This disclosure does not impose further limitations or details on the internal structure of the multimodal model; for example, it can include, but is not limited to, neural network models, deep learning models, machine learning models, or other custom network models. This disclosure does not impose further limitations or details on the internal structure of the multimodal model.

[0090] For specific implementation methods of the above image processing, please refer to the relevant descriptions of image processing in the foregoing embodiments of this disclosure, which will not be repeated here.

[0091] The following describes some optional embodiments involved in this disclosure.

[0092] In an optional embodiment, after obtaining the processing result corresponding to the second object, the present disclosure can directly display the processing result on the display screen for the user to view.

[0093] In another alternative embodiment, please refer to FIG9, which is a schematic flowchart illustrating a follow-up question answering process according to an exemplary embodiment. The process shown in FIG9 may include the following implementation steps: S901, obtaining follow-up question information of the image to be processed.

[0094] In this disclosure, the aforementioned follow-up questions can refer to questions the user wants to ask about the image to be processed. For example, referring to the example shown in Figure 6 above, the image processing is a classification and recognition process. After identifying the second object in the image to be processed as a girl, this disclosure can continue to ask follow-up questions such as the girl's height, finding similar clothes worn by the girl, etc. It can also continue to ask follow-up questions such as the height of the boy in the image to be processed; this disclosure does not limit this.

[0095] This disclosure does not limit the implementation method of obtaining the above-mentioned follow-up information. For example, the follow-up information entered by the user on the interactive interface can be directly detected and obtained. This disclosure does not impose any restrictions on this.

[0096] S902. Based on the pre-classification results corresponding to each of the m first objects in the image to be processed and / or the processing results corresponding to the second object, search for answers to the follow-up questions to obtain the corresponding follow-up questions.

[0097] This disclosure does not limit the specific implementation of the above-mentioned search and answer method. For example, historical question and answer information can be obtained and input into the corresponding multimodal model for training to obtain a trained multimodal model. The trained multimodal model is then used to search and answer the above-mentioned follow-up questions. Specifically, the multimodal model can combine the pre-classification results corresponding to each of the m first objects in the image to be processed and / or the processing results corresponding to the second objects to retrieve and answer the above-mentioned follow-up questions, thereby obtaining the corresponding follow-up answers. This disclosure does not impose further limitations or details on this.

[0098] In another optional embodiment, before using the pre-classification model in step S202, the pre-classification model needs to be trained in advance. Please refer to Figure 10, which is a flowchart illustrating a pre-classification model training method according to an exemplary embodiment. The method shown in Figure 10 can be applied to a training device, and the method may include: S1001, obtaining a training set, wherein the training set includes at least n sample images, where n is a positive integer.

[0099] The training set mentioned above in this disclosure may include information used to train the classification model. It may include at least n sample images, and optionally may also include, for example, historical question information for all or some of the sample images, corresponding historical question answers, or other custom information. This disclosure does not impose further limitations or details on this. The historical question answers may include positive sample answers and negative sample answers, that is, positive sample answers with higher accuracy and negative sample answers with lower accuracy. Taking the historical question information, such as "What kind of car is this?", as an example, the positive sample answer could be "XX brand car," and the negative sample answer could be "I drove this car to XX place yesterday," etc. This disclosure does not impose further limitations or details on this. The sample images mentioned above refer to images containing one or more objects. This disclosure does not limit the number of sample images; generally, the larger the number, the higher the accuracy of the model training. The following description uses n sample images as an example, but this does not constitute a limitation. Here, n is a positive integer that the system or user customizes according to actual conditions. For example, it may be an empirical value set based on user experience, or a statistical value calculated from a series of experimental data.

[0100] This disclosure does not limit the implementation method for obtaining the training set. For example, it can be obtained by constructing images and text, or by obtaining them from other devices or service platforms via the network. Each sample image in the training set is related to the vertical category or specific task to which the image processing belongs. Taking the image processing as classification and recognition as an example, the sample images need to include images with clearly defined classification objects, etc., which this disclosure does not limit or elaborate on further.

[0101] The training equipment mentioned in this disclosure may refer to equipment with training capabilities, including but not limited to, the aforementioned computer equipment, terminal equipment, network equipment, server equipment, or other equipment with training capabilities. This disclosure does not impose further limitations or descriptions on such equipment.

[0102] S1002. Based on the pre-classification model to be trained, pre-train and align the n sample images in the training set to obtain the alignment features corresponding to each of the n sample images.

[0103] This disclosure does not limit the specific implementation of the above-described pre-training alignment. For example, referring to the pre-classification model shown in Figure 3 above, please refer to Figure 11, which is a schematic diagram of the internal process of model training according to an exemplary embodiment. As shown in Figure 11, this disclosure can use the encoder in the pre-classification model to encode the features of n sample images respectively, thereby obtaining the image features of each of the n sample images in the image space. Further, the adapter in the pre-classification model is used to align and spatially transform the image features of each of the n sample images in the image space, converting / mapping them into the image features of each of the n sample images in the text space, thereby obtaining the alignment features corresponding to each of the n sample images.

[0104] S1003. Perform supervised fine-tuning on the alignment features corresponding to each of the n sample images to obtain the fine-tuned features corresponding to each of the n sample images.

[0105] Next, this disclosure can employ supervised learning in the adapter and large language model of the aforementioned pre-classification model, and combine user question information to fine-tune and understand the image features (i.e., the alignment features mentioned above) of each of the n sample images in the text space, thereby obtaining fine-tuned features corresponding to each of the aforementioned n sample images with higher accuracy. This disclosure does not impose further limitations or interpretations on this.

[0106] S1004. Perform preference optimization on the fine-tuning features corresponding to each of the n sample images to adjust the model parameters of the pre-classification model to be trained, thereby obtaining the trained pre-classification model.

[0107] This disclosure does not limit the specific implementation of the aforementioned preference optimization. For example, this disclosure can perform positive sample enhancement and negative sample reduction processing on the fine-tuning features corresponding to each of the n sample images based on their respective historical question information, positive sample answers, and negative sample answers. In specific implementation, this disclosure can use algorithms such as Direct Preference Optimization (DPO) to use the positive sample answers corresponding to the historical question information to perform positive sample enhancement processing on the fine-tuning features corresponding to each of the n sample images; conversely, it can use the negative sample answers corresponding to the historical question information to perform negative sample reduction processing on the fine-tuning features corresponding to each of the n sample images, thereby adjusting the model parameters of the corresponding components in the aforementioned pre-classification model, thereby obtaining the final trained pre-classification model. This disclosure does not impose further limitations or details on this.

[0108] As can be seen, the present disclosure uses pre-classification to process the second object in the image, resulting in a more intelligent and dynamic processing result. Furthermore, the simplified interactive interface provides a better user experience and supports user follow-up questions. In specific implementation, a computer device acquires an image to be processed, which includes m first objects, where m is a positive integer. A pre-classification model is called to pre-classify the m first objects, obtaining pre-classification results for each of the m first objects. These pre-classification results indicate the type of the corresponding first object. Based on the pre-classification results for each of the m first objects, image processing is performed on the second object in the image to be processed, obtaining the processing result for the second object, which is an object among the m first objects. Thus, the present disclosure pre-classifies the objects in the image before image processing, thereby improving the effect and accuracy of image processing.

[0109] In another optional embodiment, the method of acquiring the image to be processed in step S201 is not limited. For example, the terminal device can first acquire the image to be processed and then send it to a computer device (such as a server) via a network for subsequent pre-classification and image processing. Please refer to Figure 12, which is a flowchart illustrating another image processing method according to an exemplary embodiment. The method shown in Figure 12 can be applied to a terminal device, and the method may include the following implementation steps: S1201, after receiving a wake-up command from a voice assistant, displaying an interactive interface, the interactive interface including a target control, the wake-up command being used to wake up the voice assistant of the terminal device, and the target control being a functional control for acquiring an image.

[0110] In this disclosure, the wake-up command can refer to a command used to wake up a terminal device, specifically, for example, a command to wake up the voice assistant of the terminal device. This disclosure does not limit the specific form of the wake-up command; for example, it can include, but is not limited to, voice wake-up, text wake-up, manual operation wake-up (such as clicking a wake-up button), or other custom wake-up methods. This disclosure does not impose further limitations or details on these methods. The interactive interface can refer to the user interface displayed on the terminal device. This interface can include at least a target control, and optionally, other functional controls. The target control refers to a functional control used to acquire the image to be processed, which can include, but is not limited to, a camera icon, a webcam, or other similar elements.

[0111] S1202. After receiving the operation instruction for the target control, obtain the image to be processed, and perform corresponding pre-classification and image processing based on the m first objects included in the image to be processed, where m is a positive integer.

[0112] Upon receiving a user's wake-up command, this disclosure allows the display of a corresponding interactive interface on the terminal device's screen, including a target control. The user can perform specified operations on the target control, such as selection, single-click, double-click, etc. After detecting the user's specified operation on the target control, the terminal device can generate an operation instruction for that target control. Upon receiving the operation instruction for the target control, the terminal device can acquire the image to be processed corresponding to that operation instruction. For illustrative purposes, please refer to Figure 13, a schematic diagram illustrating image acquisition according to an exemplary embodiment. As shown in Figure 13, taking a camera icon as an example, after receiving the wake-up command, the terminal device can display an interactive interface including a camera icon. The user can click the camera icon to enter the shooting interface and take a picture, thereby acquiring the image to be processed. That is, after receiving the user's click instruction on the camera icon, the terminal device can take a picture and acquire the image to be processed.

[0113] Optionally, after obtaining the image to be processed, the terminal device can continue to perform pre-classification and image processing on the image. Alternatively, the terminal device can send the image to be processed to a computer device such as a server for subsequent processing, such as pre-classification and image processing of the image. Specifically, pre-classification and image processing can be performed on the m first objects in the image, which can be referred to the relevant descriptions in the embodiments described in Figures 2-11 above, and will not be repeated here.

[0114] In an optional embodiment, the user can also perform a specified operation on any first object in the image to be processed, such as circling, clicking, drawing, double-clicking, or other operations. Correspondingly, the terminal device can detect and receive the operation instruction for any first object in the image to be processed. Then, in response to the operation instruction for any first object, the user can identify the first object as a second object. This facilitates subsequent image processing of the second object in the image to be processed, which can be referred to in the relevant descriptions of the foregoing embodiments, and will not be repeated here. For example, referring to the example shown in Figure 6 above, the user selects any first object (the girl) in the image to be processed as the second object through a circling operation, which can be referred to in the relevant descriptions of the example shown in Figure 6 above, and will not be repeated here.

[0115] By implementing the embodiments of this disclosure, after receiving a wake-up command from a voice assistant, the terminal device displays an interactive interface. The interactive interface includes a target control, and the wake-up command is used to wake up the terminal device's voice assistant. After receiving an operation command for the target control, an image to be processed is acquired, and pre-classification and image processing are performed based on m first objects included in the image to be processed, where m is a positive integer. Thus, after waking up the terminal device's voice assistant, the image to be processed can be acquired through a simplified interactive interface, facilitating subsequent image processing of the image. This improves the user experience.

[0116] Based on the foregoing embodiments, please refer to FIG14, which is a schematic diagram of an image processing apparatus according to an exemplary embodiment. The apparatus shown in FIG14 can be applied in a computer device, and the apparatus may include an acquisition module 1401 and a processing module 1402. Specifically: the acquisition module 1401 is configured to acquire an image to be processed, the image to be processed including m first objects, where m is a positive integer; the processing module 1402 is configured to call a pre-classification model to pre-classify the m first objects, obtaining pre-classification results corresponding to each of the m first objects, the pre-classification results indicating the type to which the corresponding first object belongs; the processing module 1402 is further configured to perform image processing on a second object in the image to be processed based on the pre-classification results corresponding to each of the m first objects, obtaining a processing result corresponding to the second object, where the second object is one of the m first objects.

[0117] In some embodiments, the processing module 1402 is configured to: determine the priority of the second object in the image to be processed; perform image processing on the second object based on the pre-classification result of the second object and the priority of the second object to obtain the processing result.

[0118] In some embodiments, the processing module 1402 is configured to: after receiving an operation instruction for any of the first objects, determine any of the first objects as the second objects; and determine the priority of the second objects as a preset priority.

[0119] In some embodiments, the processing module 1402 is configured to: calculate the proportion of m first objects in the image to be processed to obtain the proportion of each of the m first objects; and determine the second object and the priority of the second object based on the proportion of each of the m first objects; wherein the priority of the second object and the proportion of the second object are positively correlated.

[0120] In some embodiments, the processing module 1402 is configured to: acquire user question information; perform intent understanding and priority determination on m first objects in the image to be processed based on the user question information, and obtain the second object and the priority of the second object.

[0121] In some embodiments, the acquisition module 1401 is further configured to acquire a training set, the training set including at least n sample images, where n is a positive integer; the processing module 1402 is further configured to perform pre-training alignment on the n sample images in the training set based on the pre-classification model to be trained, to obtain alignment features corresponding to each of the n sample images; the processing module 1402 is further configured to perform supervised instruction fine-tuning on the alignment features corresponding to each of the n sample images, to obtain fine-tuned features corresponding to each of the n sample images; the processing module 1402 is further configured to perform preference optimization on the fine-tuned features corresponding to each of the n sample images, to adjust the model parameters of the pre-classification model to be trained, thereby obtaining the trained pre-classification model.

[0122] In some embodiments, the processing module 1402 is configured to: encode the image features of the n sample images in the training set based on the pre-classification model to be trained, to obtain the image features of each of the n sample images in the image space; align and spatially transform the image features of each of the n sample images in the image space to obtain the alignment features corresponding to each of the n sample images; wherein, the alignment features are the image features of the corresponding sample images in the text space.

[0123] In some embodiments, the training set further includes historical question information and historical question answers for the sample images, wherein the historical question answers include positive sample answers and negative sample answers, and the processing module 1402 is configured to perform positive sample enhancement and negative sample reduction processing on the fine-tuning features corresponding to each of the n sample images based on the historical question information, positive sample answers, and negative sample answers of each of the n sample images.

[0124] In some embodiments, the acquisition module 1401 is further configured to acquire follow-up information of the image to be processed; the processing module 1402 is further configured to search for answers to the follow-up information based on the pre-classification results corresponding to each of the m first objects in the image to be processed and / or the processing results corresponding to the second objects, and obtain corresponding follow-up answers.

[0125] For any content not introduced or described in the embodiments of this disclosure, please refer to the relevant descriptions in the foregoing method embodiments; they will not be repeated here.

[0126] By implementing the embodiments of this disclosure, the above-described apparatus can acquire an image to be processed, the image to be processed including m first objects, where m is a positive integer; call a pre-classification model to pre-classify the m first objects, obtaining pre-classification results corresponding to each of the m first objects, the pre-classification results being used to indicate the type to which the corresponding first object belongs; based on the pre-classification results corresponding to each of the m first objects, perform image processing on a second object in the image to be processed, obtaining a processing result corresponding to the second object, where the second object is an object among the m first objects. Thus, this disclosure allows for pre-classification of objects in the image to be processed before image processing, thereby improving the effect and accuracy of image processing.

[0127] Please refer to Figure 15, which is a schematic diagram of another image processing apparatus according to an exemplary embodiment. The apparatus shown in Figure 15 can be applied to a computer device and may include a processing module 1501 and an acquisition module 1502. Specifically: the processing module 1501 is configured to display an interactive interface after receiving a wake-up command from a voice assistant. The interactive interface includes a target control, and the wake-up command is used to wake up the voice assistant of the terminal device. The target control is a functional control for acquiring an image. The acquisition module 1502 is configured to acquire an image to be processed after receiving an operation command for the target control, and to perform corresponding pre-classification and image processing based on m first objects included in the image to be processed, where m is a positive integer.

[0128] In some embodiments, the acquisition module 1502 is further configured to receive an operation instruction for any of the first objects in the image to be processed; the processing module 1501 is further configured to, in response to the operation instruction for any of the first objects, determine any of the first objects as second objects, so as to perform image processing on the second objects.

[0129] For any content not introduced or described in the embodiments of this disclosure, please refer to the relevant descriptions in the foregoing method embodiments; they will not be repeated here.

[0130] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the image processing method provided in this disclosure.

[0131] Figure 16 is a schematic diagram illustrating the structure of another image processing apparatus according to an exemplary embodiment. For example, the image processing apparatus 500 described above can be a computer device or a terminal device, which can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0132] Referring to FIG16, the image processing device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output interface 512, sensor component 514, and communication component 516.

[0133] Processing component 502 typically controls the overall operation of device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the image processing method described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0134] Memory 504 is configured to store various types of data to support the operation of device 500. Examples of such data include instructions for any application or method operating on device 500, contact data, phonebook data, messages, pictures, videos, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0135] Power supply component 506 provides power to various components of device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0136] Multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the image processing device 500 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0137] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0138] Input / output interface 512 provides an interface between processing component 502 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0139] Sensor assembly 514 includes one or more sensors for providing status assessments of various aspects of device 500. For example, sensor assembly 514 may detect the on / off state of device 500, the relative positioning of components such as the display and keypad of device 500, changes in the position of device 500 or a component of device 500, the presence or absence of user contact with device 500, the orientation or acceleration / deceleration of device 500, and temperature changes of device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0140] Communication component 516 is configured to facilitate wired or wireless communication between device 500 and other devices. Device 500 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0141] In an exemplary embodiment, the apparatus 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method described above.

[0142] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by the processor 520 of the device 500 to complete the above-described upper-level image processing method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0143] The aforementioned device can be a standalone electronic device or a part of a standalone electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be a single IC or a collection of multiple ICs. The chip can include, but is not limited to, the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), and SoC (System on Chip). The aforementioned integrated circuit or chip can be used to execute executable instructions (or code) to implement the aforementioned image processing method. The executable instructions can be stored in the integrated circuit or chip or obtained from other devices or equipment. For example, the integrated circuit or chip includes a processor, memory, and an interface for communicating with other devices. The executable instructions can be stored in the memory, and when the executable instructions are executed by the processor, the above-described image processing method can be implemented; alternatively, the integrated circuit or chip can receive the executable instructions through the interface and transmit them to the processor for execution to implement the above-described image processing method.

[0144] For any content not introduced or described in the embodiments of this disclosure, please refer to the relevant descriptions in the foregoing method embodiments, which will not be repeated here.

[0145] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the image processing method described above when executed by the programmable device.

[0146] Please refer to Figure 17, which is a schematic diagram of a chip structure according to an exemplary embodiment. The chip 600 shown in Figure 17 includes a processor 601 and an interface 602. Optionally, it may also include a memory 603. The number of processors 601 can be one or more, and the number of interfaces 602 can be multiple.

[0147] In one embodiment, for a chip used to implement the method embodiments of this disclosure: the interface 602 is used to receive or output signals; the processor 601 is used to execute some or all of the contents of the image processing method embodiments.

[0148] Understandably, the processor in this embodiment of the disclosure can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiment can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0149] Understandably, the memory in the embodiments of this disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0150] It should be noted that the descriptions of the storage media, devices, and chip embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage media, storage media, and device embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0151] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0152] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain an image to be processed, wherein the image to be processed includes m first objects, where m is a positive integer; The pre-classification model is invoked to pre-classify m first objects, and the pre-classification results corresponding to each of the m first objects are obtained. The pre-classification results are used to indicate the type to which the corresponding first object belongs. Based on the pre-classification results corresponding to each of the m first objects, image processing is performed on the second object in the image to be processed to obtain the processing result corresponding to the second object, wherein the second object is an object among the m first objects.

2. The method according to claim 1, characterized in that, The step of performing image processing on the second object in the image to be processed based on the pre-classification results corresponding to each of the m first objects to obtain the processing result corresponding to the second object includes: determining the priority of the second object in the image to be processed; and performing image processing on the second object based on the pre-classification result and the priority of the second object to obtain the processing result.

3. The method according to claim 2, characterized in that, Determining the priority of the second object in the image to be processed includes: after receiving an operation instruction for any first object, identifying the any first object as the second object; and determining the priority of the second object as a preset priority.

4. The method according to claim 2, characterized in that, Determining the priority of the second object in the image to be processed includes: calculating the proportion of each of the m first objects in the image to be processed to obtain the proportion of each of the m first objects; determining the second object and its priority based on the proportion of each of the m first objects; wherein the priority of the second object and its proportion are positively correlated.

5. The method according to claim 2, characterized in that, Determining the priority of the second object in the image to be processed includes: obtaining user question information; performing intent understanding and priority determination on m first objects in the image to be processed based on the user question information to obtain the second object and its priority.

6. The method according to claim 1, characterized in that, The method further includes: acquiring a training set, the training set including at least n sample images, where n is a positive integer; pre-training and aligning the n sample images in the training set based on the pre-classification model to be trained, to obtain alignment features corresponding to each of the n sample images; performing supervised fine-tuning on the alignment features corresponding to each of the n sample images, to obtain fine-tuned features corresponding to each of the n sample images; and performing preference optimization on the fine-tuned features corresponding to each of the n sample images to adjust the model parameters of the pre-classification model to be trained, thereby obtaining the trained pre-classification model.

7. The method according to claim 6, characterized in that, The pre-training alignment of the n sample images in the training set based on the pre-classification model to be trained, to obtain the alignment features corresponding to each of the n sample images, includes: encoding the image features of the n sample images in the training set based on the pre-classification model to be trained, to obtain the image features of each of the n sample images in the image space; aligning and spatially transforming the image features of each of the n sample images in the image space to obtain the alignment features corresponding to each of the n sample images; wherein, the alignment features are the image features corresponding to the sample images in the text space.

8. The method according to claim 6, characterized in that, The training set also includes historical question information and historical question answers for the sample images. The historical question answers include positive sample answers and negative sample answers. The preference optimization of the fine-tuning features corresponding to each of the n sample images includes: based on the historical question information, positive sample answers, and negative sample answers of each of the n sample images, performing positive sample enhancement and negative sample reduction processing on the fine-tuning features corresponding to each of the n sample images.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: obtaining follow-up information of the image to be processed; and searching for answers to the follow-up information based on the pre-classification results corresponding to each of the m first objects in the image to be processed and / or the processing results corresponding to the second objects, to obtain the corresponding follow-up answers.

10. An image processing method, characterized in that, include: Upon receiving a wake-up command from the voice assistant, an interactive interface is displayed. The interactive interface includes a target control. The wake-up command is used to wake up the voice assistant of the terminal device. The target control is a functional control for acquiring images. Upon receiving an operation command for the target control, an image to be processed is acquired. Based on m first objects included in the image to be processed, corresponding pre-classification and image processing are performed, where m is a positive integer.

11. The method according to claim 10, characterized in that, The method further includes: receiving an operation instruction for any of the first objects in the image to be processed; and, in response to the operation instruction for any of the first objects, determining any of the first objects as a second object, so as to perform image processing on the second object.

12. An image processing apparatus, characterized in that, include: The acquisition module is configured to acquire an image to be processed, wherein the image to be processed includes m first objects, where m is a positive integer; The processing module is configured to call a pre-classification model to pre-classify m of the first objects, and obtain the pre-classification results corresponding to each of the m first objects. The pre-classification results are used to indicate the type to which the corresponding first object belongs. The processing module is further configured to perform image processing on the second object in the image to be processed based on the pre-classification results corresponding to each of the m first objects, and obtain the processing result corresponding to the second object, wherein the second object is an object among the m first objects.

13. An image processing apparatus, characterized in that, include: The processing module is configured to display an interactive interface after receiving a wake-up command from the voice assistant. The interactive interface includes a target control. The wake-up command is used to wake up the voice assistant of the terminal device. The target control is a functional control for acquiring images. The acquisition module is configured to acquire an image to be processed after receiving an operation command for the target control, so as to perform corresponding pre-classification and image processing based on m first objects included in the image to be processed, where m is a positive integer.

14. A computer device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the steps of the method according to any one of claims 1 to 9; or, the processor is configured to execute the executable instructions to implement the steps of the method according to any one of claims 10 or 11.

15. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9; or, when the computer program instructions are executed by the processor, they implement the steps of the method according to any one of claims 10 or 11.

16. A chip, characterized in that, It includes a processor and an interface; the processor is configured to read instructions to execute the method of any one of claims 1 to 9; or, the processor is configured to read instructions to execute the method of any one of claims 10 or 11.

Images