Image processing method and device and related equipment

Abstract

The embodiment of the invention discloses an image processing method and device and related equipment, and can be applied to the technical field of data processing. The method comprises the following steps: acquiring a business image to be subjected to image quality detection; the business image is an image in a target business scene; retrieving a target standard image matched with the business image in a standard image set corresponding to the target business scene, and obtaining target image quality data bound with the target standard image; when the business feature data and the target standard feature data are obtained, splicing the business feature data, the target standard feature data and the target image quality data to obtain multi-modal feature data; and when model text prompt information for the target business model is obtained, the target business model performs image quality detection on the multi-modal feature data according to the model text prompt information to obtain image quality detection data. By adopting the embodiment of the invention, the accuracy and flexibility of image quality detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to image processing methods, devices and related equipment. Background Art

[0002] In the prior art, in different business scenarios, there will be different image quality judgment standards for images in different business scenarios. Therefore, it is necessary to construct different neural network models for images in different business scenarios to perform image quality detection on the images. For example, it is necessary to construct a neural network model corresponding to a certain image quality judgment standard (for example, network model Mo1) for images in a certain business scenario (for example, B1 business scenario), and to construct a neural network model corresponding to another image quality judgment standard (for example, network model Mo2) for images in another business scenario (for example, B2 business scenario).

[0003] However, the inventors discovered in practice that, based on the existing image quality judgment standards, it is impossible to use the same neural network model to flexibly implement quality detection of images in different business scenarios. For example, if the network model Mo1 corresponding to the B1 business scenario is used to perform quality detection on images in another business scenario (for example, the B2 business scenario), it will be difficult to accurately obtain quality detection results that match the images in the B2 business scenario due to the insufficient generalization ability of the network model Mo1, thereby reducing the accuracy of image quality detection. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus and related equipment, which can retrieve a target standard image that matches a business image in a target business scenario to which the business image belongs, thereby fusing feature data and image quality data of the target standard image to perform image quality detection, and can obtain an image quality detection result that is more suitable for the image quality judgment standard of the target business scenario, thereby improving the accuracy of image quality detection.

[0005] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire a business image to be subjected to image quality detection; the business image is an image in a target business scenario; the target business scenario refers to a scene in at least one business scenario; one business scenario corresponds to one standard image set;

[0007] In the standard image set corresponding to the target business scenario, a target standard image matching the business image is retrieved, and target image quality data bound to the target standard image is obtained;

[0008] When the business feature data of the business image and the target standard feature data of the target standard image are acquired, the business feature data, the target standard feature data and the target image quality data are spliced ​​to obtain multimodal feature data for inputting into the target business model;

[0009] When the model text prompt information for the target business model is obtained, the multimodal feature data and the model text prompt information are input into the target business model, and the target business model performs image quality detection on the multimodal feature data according to the model text prompt information to obtain image quality detection data.

[0010] On the one hand, an embodiment of the present application provides an image processing device, the device comprising:

[0011] An image acquisition unit is used to acquire a business image to be subjected to image quality detection; the business image is an image in a target business scenario; the target business scenario refers to a scene in at least one business scenario; one business scenario corresponds to one standard image set;

[0012] An image retrieval unit, used to retrieve a target standard image matching the business image from a standard image set corresponding to a target business scenario, and obtain target image quality data bound to the target standard image;

[0013] A feature data splicing unit is used to splice the business feature data, the target standard feature data and the target image quality data to obtain multimodal feature data for inputting into the target business model when the business feature data of the business image and the target standard feature data of the target standard image are acquired;

[0014] The image quality detection unit is used to input the multimodal feature data and the model text prompt information into the target business model when the model text prompt information for the target business model is obtained, and the target business model performs image quality detection on the multimodal feature data according to the model text prompt information to obtain image quality detection data.

[0015] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method provided by the embodiment of the present application.

[0016] In one aspect, the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the present application.

[0017] In an embodiment of the present application, a target standard image matching the business image can be retrieved from a standard image set corresponding to a target business scene to which the business image belongs, thereby splicing the image quality data corresponding to the target standard image (i.e., target image quality data), the standard feature data corresponding to the target standard image (i.e., target standard feature data), and the business feature data corresponding to the business image to obtain multimodal feature data, so that the target business model can perform image quality detection on the multimodal feature data to obtain image quality detection data. Among them, the target standard image matching the business image retrieved from the standard image set corresponding to the target business scene is equivalent to a reference for the image quality judgment standard when performing image quality detection, so that when performing image quality detection on the business image data under any business scene (such as the target business scene), the target standard image matching the business image under the corresponding business scene can be input into the target business model for processing, so that the target business model can automatically mine the image quality judgment standard of the current target business scene, so that the image quality detection data that is more suitable for the image quality judgment standard of the target business scene to which the business image belongs can be obtained, thereby improving the accuracy of image quality detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a structural diagram of a data processing system provided in an embodiment of the present application;

[0020] Figure 2 is a scene schematic diagram of an image processing method provided in an embodiment of the present application;

[0021] Figure 3 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0022] Figure 4 It is a flowchart of determining a target business scenario provided by an embodiment of the present application;

[0023] Figure 5 is a flowchart of an image quality detection process provided by an embodiment of the present application;

[0024] Figure 6 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0025] Figure 7 It is a flowchart of a feature extraction process provided by an embodiment of the present application;

[0026] Figure 8 It is a flowchart of a training process of a semantic feature extractor provided in an embodiment of the present application;

[0027] Fig. 9 It is a flowchart of a training process of a quality feature extractor provided in an embodiment of the present application;

[0028] Fig.10 is a flowchart of another image quality detection process provided by an embodiment of the present application;

[0029] Fig.11 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0030] Fig.12 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] See also Figure 1 , Figure 1 Schematic diagram of a data processing system provided in an embodiment of the present application. Figure 1 As shown, the data processing system may include terminal devices (such as device 11a, device 12a, device 13a) and server 100a. It can be understood that Figure 1 The number of terminal devices and servers in the embodiment is only for illustration, and any number of terminal devices and servers may be provided according to the implementation requirements. The terminal devices (such as the device 11a, the device 12a, and the device 13a) may communicate with the server through a network (i.e., a medium providing a communication link through a wired or wireless communication link or an optical fiber cable, etc.) to transmit data.

[0033] It is understandable that a client may be running on a terminal device (such as device 12a), and the client may be a program that provides local services to a user (also referred to as a target object or an operation object). Server 100a may be a server corresponding to a client, and a program for providing resources, service data and other services to the client may be running in the server 100a. It is understandable that a client running on a terminal device may also be referred to as an application client, a business client, and the like. For example, the client running on the terminal device may be a client that involves viewing images or videos (such as a client of a social application, a client of an e-commerce platform, a video client, and the like), and then a user may publish or upload images or videos through the client, and then the server may perform image quality detection on the images or videos uploaded by the user, and perform corresponding business processing on the images uploaded by the user based on the image quality detection results, for example, when it is detected that the image quality of the images or videos uploaded by the user is unqualified, the probability of recommending the images or videos uploaded by the user is reduced.

[0034] It is understandable that terminal devices (such as device 11a) may include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, smart speakers, smart home appliances, etc., without limitation here. The server 100a may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, without limitation here. The embodiments of the present application may be applied to the above-mentioned terminal devices, or to the above-mentioned servers, without limitation here.

[0035] For further information, see Figure 2 , Figure 2 Schematic diagram of a scene of an image processing method provided in an embodiment of the present application. Figure 2 As shown, firstly, the business image It to be subjected to image quality detection can be obtained (such as Figure 2 As shown in 211a in FIG. 1 , the business image may be an image uploaded or published by a business object (such as a user), or a video frame in a business video uploaded or published by a business object (such as a user). The target business scene corresponding to the business image It may be determined from at least one business scene (step S21a), wherein at least one business scene may refer to a scene requiring image quality detection, such as Figure 2As shown in 2111a in FIG. 1 , at least one business scenario may include business scenario C1, business scenario C2, and business scenario C3, etc. For example, business scenario C1 may be a resource recommendation scenario, business scenario C2 may be a news scenario, and business scenario C3 may be a social application scenario. A corresponding standard image set may be configured for each business scenario, and each standard image set may include multiple standard images. A standard image may refer to an image used as a reference when performing image quality detection on a business image. For example, Figure 2 As shown in 2112a, the business scenario C1 has a corresponding standard image set J1, the business scenario C2 has a corresponding standard image set J2, and the business scenario C3 has a corresponding standard image set J3.

[0036] Further, after determining that the target business scene corresponding to the business image It is business scene C1 (eg Figure 2 212a in FIG. 2 ), obtain the standard image set J1 corresponding to the business scene C1 (as shown in FIG. ... Figure 2 213a in FIG), and then perform image retrieval in the standard image set J1 (step S22a), so that a target standard image (such as Figure 2 The target standard image may include multiple standard images. For example, the target standard image may be M standard images with the largest image similarity to the target business image, such as Figure 2 As shown in 2141a in FIG. 1 , the target standard image may include: standard image Itop1, standard image Itop2, ..., standard image ItopM, etc. It can be understood that each standard image may be associated with corresponding image quality data, and the image quality data may be data used to represent the image quality of the standard image, for example, the image quality data is an image quality score. For example, Figure 2 As shown in 2142a, the standard image Itop1 is bound to the image quality data Qtop1, the standard image Itop2 is bound to the image quality data Qtop2, and so on, the standard image ItopM is bound to the image quality data QtopM.

[0037] Furthermore, the target standard feature data corresponding to the target standard image (such as Figure 2 The standard feature data corresponding to any standard image may be obtained by performing feature extraction processing on the corresponding standard image. Furthermore, the target image quality data (such as Figure 2 216a in FIG. 1 ).

[0038] Furthermore, feature extraction processing can be performed on the business image It (step S23a) to obtain business feature data corresponding to the business image It (such as Figure 2 Further, the service feature data, the target standard feature data and the target image quality data may be spliced ​​to obtain multi-modal feature data (such as Figure 2 As shown in 218a), the business modality corresponding to the business feature data and the target standard feature data is the image feature modality, and the business modality of the target image quality data is the text modality, so that the multimodal feature data can be composed based on data of different business modalities.

[0039] Furthermore, you can obtain model text prompt information (such as Figure 2 The model text prompt information may be prompt information for prompting the target business model on a method for processing the multimodal feature data. Then, the model text prompt information and the multimodal feature data are input into the target business model (eg Figure 2 For example, the target service model may be a multimodal large language model (i.e., MLLM). Further, the target service model may perform image quality detection on the multimodal feature data according to the model text prompt information to obtain image quality detection data (e.g., Figure 2 The image quality detection data may be data obtained by performing quality detection on the business image. The image quality detection data may be expressed as an image quality score of the business image.

[0040] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of the relevant regions.

[0041] It is understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of the present application. The technical solutions of the present application can also be applied to other scenarios. For example, it is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0042] For further information, see Figure 3 , Figure 3 is a flowchart of an image processing method provided by an embodiment of the present application. The method can be executed by a computer device, for example, the computer device can be the above-mentioned Figure 1 The method may at least include the following steps S101 to S104.

[0043] S101. Acquire a business image to be subjected to image quality detection; the business image is an image in a target business scenario; the target business scenario refers to a scene in at least one business scenario; and one business scenario corresponds to one standard image set.

[0044] The business image may refer to an image to be subjected to image quality detection. The business image may be an image in a target business scenario. In the embodiment of the present application, the image quality of the business image may be reflected in the clarity, noise information, etc. of the business image.

[0045] Among them, the target business scenario may refer to the business scenario to which the business image to be subjected to image quality detection belongs. The target business scenario may refer to any business scenario of at least one business scenario, and the at least one business scenario may refer to a scenario in which image quality detection is required. For ease of description, the at least one business scenario may be referred to as a business scenario set, which may be configured by relevant management personnel. For example, the business scenarios in the business scenario set may be resource recommendation scenarios, news scenarios, social application scenarios, etc., which are not limited here. In these scenarios, image quality detection may be performed on the image. If it is detected that the image quality of the image is unqualified (that is, the image quality detection result of the business image indicates that the image quality of the business image is unqualified), the business image may be subjected to corresponding business processing. For example, in a resource recommendation scenario, if it is detected that the image quality of the business image in the image resource is unqualified, the probability of recommending the image resource is reduced, thereby avoiding the push of low-quality image resources and improving the overall quality of image resources in the application.

[0046] In one embodiment, when a business image is acquired, the target business scene to which the business image belongs can be determined based on the target scene identifier associated with the business image. Specifically, the target scene identifier associated with the business image is acquired, a business scene matching the target scene identifier is searched from at least one business scene, and the business scene matching the target scene identifier is determined as the target business scene. The target scene identifier may refer to a scene identifier corresponding to the business scene to which the business image belongs, and one scene identifier is used to uniquely identify one business scene.

[0047] For example, see Figure 4 , Figure 4 is a flow chart of determining a target business scenario provided by an embodiment of the present application. Figure 4 As shown, the acquired business image (such as Figure 4 401a in FIG. 401a) is associated with a scene identifier Cb1 (such as Figure 4 Then, the business scenario set 403a can be searched for a business scenario that matches the scenario identifier Cb1 (step S41a). It can be seen that the business scenario set 403a can include the business scenario C1 (as shown in 402a in FIG. 4 ). Figure 4 404a in ), business scenario C2 (as shown in Figure 4 As shown in 405a), ....Business scenario Cn (such as Figure 4 406a in FIG. 4 ), each business scenario is associated with a corresponding scenario identifier. For example, business scenario C1 is associated with a scenario identifier Cb1 (as shown in FIG. 406a in FIG. 4 ). Figure 4 414a in FIG), the business scenario C2 is associated with a scenario identifier Cb2 (eg Figure 4 415a in FIG. 4 ), and so on, the business scenario Cn is associated with a scenario identifier Cbn (such as Figure 4 Based on this, it can be seen that the business image (such as Figure 4 As shown in 401a in FIG. 4 ), the business scenario that matches the scenario identifier Cb1 is the business scenario C1, so that the business scenario C1 can be determined as the target business scenario (eg, Figure 4 407a in FIG. 1 ).

[0048] It can be understood that each business scenario has a corresponding standard image set, that is, one business scenario corresponds to one standard image set, and one standard image set may include multiple standard images, and a standard image may refer to an image used as a reference when performing image quality detection on a business image. The standard image sets corresponding to each business scenario belong to the standard image library, and when the business image under the target business scenario can be determined, the standard image set corresponding to the target business scenario can be found from the standard image library.

[0049] In a standard image set (or a standard image library), each standard image may be bound to corresponding image quality data, and the image quality data may be data used to characterize the image quality of the standard image. The image quality data may be expressed as an image quality score range or an image quality score of the standard image. The image quality score may refer to a score used to represent the image quality of an image, and the image quality score range may refer to a score range used to represent the image quality of an image. For example, the image quality score of a standard image may range from 0 to 10. Then, the image quality data of any standard image may be an image quality score or an image quality score range belonging to 0 to 10. For example, the image quality score indicated by the image quality data of a standard image is 4, or the image quality score range indicated by the image quality data of a standard image is (6,8).

[0050] For example, the following table (1) is an example of a standard image set corresponding to a business scenario and its related information.

[0051] Table (1)

[0052]

[0053]

[0054] The image quality score refers to the quality score of the image, and its value range can be 0-10. The image quality category can be a category divided based on the image quality score. For example, if the value range of the image quality score of any image is 0-10, then when the image quality score belongs to the range corresponding to [1,2), the image quality category of the business image is determined to be a completely high-blur quality category; when the image quality score belongs to the range corresponding to [2,3), the image quality category of the business image is determined to be a main high-blur quality category; when the image quality score belongs to the range corresponding to [3,4), the image quality category of the business image is determined to be a fuzzy quality category; when the image quality score is 4, the image quality category of the business image is determined to be a slightly fuzzy quality category; when the image quality score belongs to the range corresponding to (4,10), the image quality category of the business image is determined to be a clear category.

[0055] Among them, the disposal result may be the disposal result of images of each image quality category under the corresponding business scenario (such as the target business scenario). For example, the business image (or the business video corresponding to the business image) may be disposed accordingly. For example, in the resource recommendation scenario, the business image to which the business image belongs may be intercepted or not intercepted. Interception here may refer to reducing the recommendation probability of the business image to which the business image belongs, and non-interception may refer to not reducing the recommendation probability of the business image to which the business image belongs.

[0056] It is understandable that each image quality category has a corresponding plurality of standard images, so each standard image corresponding to an image quality category may be bound with a corresponding image quality score and image quality category (i.e., image quality data). It is understandable that in a standard image set corresponding to a business scenario, relevant description information for each image quality category may also be included. For example, the relevant description information may include keyword information, which may describe the key description information of the corresponding image quality category (used to describe the image quality judgment standard of the corresponding image quality category). For example, the key description information for the completely high-blur quality category may be: pure color, unable to identify the subject; the key description information for the subject high-blur quality category may be: there is a subject, but the subject outline is not clear; the key description information for the fuzzy quality category may be: the subject and the outline can be distinguished, without an obvious feeling of being blurred; the key description information for the slightly blurred quality category may be: the subject is clear, and there are flaws in the details; the key description information for the clear category may be: clear. The relevant description information may also include image detail information, which may be used to specifically describe the image quality judgment standard of the corresponding image quality category. No further description is given here.

[0057] Optionally, in the embodiment of the present application, the business image may be an image directly published or uploaded by a business object (such as a user), or may be a video frame selected from a plurality of video frames included in a business video, which is not limited here. The business video may be a video published or uploaded by a business object (such as a user).

[0058] S102: Retrieve a target standard image that matches the business image from a standard image set corresponding to the target business scenario, and obtain target image quality data bound to the target standard image.

[0059] The standard image set corresponding to the target business scenario may refer to a standard image set corresponding to the target business scenario, and the standard image may refer to an image used as a reference when performing image quality detection on a business image.

[0060] It can be understood that in the embodiments of the present application, there are differences in the image quality standards for images in different business scenarios. Therefore, different standard images can be configured as references for different business scenarios to characterize images of different image quality categories (such as completely blurry, mainly blurry, slightly blurred, clear, etc.) in different business scenarios, so that the standard images belonging to the same scene as the business images can be used as a reference for image quality detection of the business images in the future, thereby obtaining image quality detection results suitable for the business scenarios to which the business images belong.

[0061] The target standard image may refer to a target standard image that matches the business image in a standard image set corresponding to the target business scenario. Specifically, the target standard image may be an image in the standard image set whose image similarity with the business image satisfies a similarity condition.

[0062] In one embodiment, in a standard image set corresponding to a target business scene, retrieving a target standard image that matches a business image can include the following steps: ① performing feature extraction processing on the business image through a feature extractor to obtain business feature data of the business image; ② obtaining standard feature data corresponding to each standard image in the standard image set corresponding to the target business scene; the standard feature data corresponding to any standard image is obtained by performing feature extraction processing on any standard image through a feature extractor; ③ based on the feature similarity between the business feature data and the standard feature data corresponding to each standard image, determining the image similarity between each standard image and the business image; ④ from the standard image set corresponding to the target business scene, searching for a standard image whose image similarity with the business image meets the similarity condition, and determining the found standard image as the target standard image that matches the business image.

[0063] Among them, the feature extractor can be a processing module for extracting features from business images. When the feature extractor performs feature extraction processing on business images, it integrates the semantic features and image quality features of the business images, so that the target standard image found is more similar to the business image, so that the standard image with high similarity to the target standard image can be used as a reference for image quality detection of the business image in the future, so that the image quality detection result of the business image is more suitable for the corresponding business scenario, which helps to improve the flexibility and accuracy of image quality detection.

[0064] The service feature data refers to feature data obtained by performing feature extraction processing on the service image. The service feature data can be represented as a feature vector, which is not limited here.

[0065] Among them, the standard feature data may refer to the feature data obtained by performing feature extraction processing on any standard image, and the standard feature data may be expressed as a feature vector, which is not limited here. It is understandable that the standard feature data corresponding to any standard image is obtained by performing feature extraction processing on any standard image by a feature extractor, and its specific extraction process can refer to the relevant description of determining the business feature data, which is not repeated here. It is understandable that in an embodiment of the present application, before retrieving a standard image matching a business image, feature extraction processing can be performed on each standard image in advance to obtain the corresponding standard feature data, and then when retrieving a standard image matching a business image, the standard feature data of each standard image can be directly obtained, thereby improving the efficiency of image retrieval, and the standard feature data of each standard image generated in advance can be stored, and then when retrieving different business images, there is no need to perform repeated calculations, and the standard feature data of each standard image generated in advance can be directly obtained, which greatly improves the efficiency of image retrieval.

[0066] Among them, feature similarity may refer to the similarity between business feature data and any standard feature data. Image similarity may refer to the similarity between a business image and any standard image. The image similarity between any standard image and a business image may be determined based on the feature similarity between the business feature data of the business image and the standard feature data of the standard image. For example, the feature similarity may be directly used as the image similarity, or the feature similarity may be converted to a similarity range (such as 0-1) corresponding to the image similarity to obtain the image similarity. It should be understood that the image similarity is proportional to the feature similarity.

[0067] Among them, the feature similarity between the business feature data and the standard feature data corresponding to any standard image may include: obtaining a feature product based on the product between the business feature data and the standard feature data corresponding to any standard image, obtaining a feature norm product based on the product between the norm of the business feature data and the norm of the standard feature data corresponding to any standard image, and then dividing the feature product by the feature norm product to obtain the feature similarity between the business feature data and the standard feature data corresponding to any standard image.

[0068] For example, the process of calculating the image similarity between a business image and any standard image can be calculated by referring to the following formula (1).

[0069]

[0070] Among them, S(I t ,I i ) represents business image I t With any standard image Ii The image similarity between them. t ) represents business image I t Business characteristic data, F(I i ) represents any standard image I i Standard feature data.

[0071] Among them, the similarity condition may refer to the conditions that the target standard image that matches the business image needs to meet. For example, the similarity condition may refer to: when each standard image is sorted from high to low according to the image similarity, the sorting number is less than or equal to the preset sorting number. Among them, the sorting number may refer to the number when each standard image is sorted, for example, the sorting number of the first standard image is 1, and the higher the ranking, the smaller the sorting number. The preset sorting number may refer to the number threshold that the sorting number of the image similarity of the target standard image needs to reach when determining the target standard image. For example, the preset sorting number may be 5, which means that the 5 standard images with the largest image similarity meet the similarity condition, that is, the 5 standard images with the largest image similarity can be selected as the target standard image. Optionally, the similarity condition may also be that the image similarity is greater than or equal to the similarity threshold. The similarity threshold may refer to the threshold that the image similarity of the target standard image needs to reach when determining the target standard image. For example, the similarity threshold may be 0.6. Alternatively, the similarity condition may be to satisfy both of the above conditions at the same time, that is, when sorting each standard image from high to low according to image similarity, the sorting number is less than or equal to the preset sorting number, and the image similarity is greater than or equal to the similarity threshold, or other similarity conditions may be used, which are not limited here. It is understandable that in the embodiment of the present application, the greater the image similarity between the business image and the standard image, the more similar the standard image is to the business image, and then based on the judgment based on the similarity image condition, the standard image that is more similar to the business image is selected as a reference to perform image quality detection on the business image, and the image quality detection of the business image is more accurate.

[0072] The target image quality data may refer to the image quality data bound to the target standard image. For the introduction of the image quality data, reference may be made to the above-mentioned related description, which will not be elaborated here.

[0073] It can be understood that the above-mentioned retrieval of the target standard image matching the business image from the standard image set corresponding to the target business scene may be performed by an image retrieval module.

[0074] S103. When the business feature data of the business image and the target standard feature data of the target standard image are obtained, the business feature data, the target standard feature data and the target image quality data are spliced ​​to obtain multimodal feature data for inputting into the target business model.

[0075] The process of obtaining the service characteristic data may refer to the above-mentioned related description and will not be elaborated here.

[0076] The target standard feature data refers to the standard feature data of the target standard image, and the acquisition process thereof can refer to the above-mentioned related description, which will not be elaborated here.

[0077] Among them, the multimodal feature data can be feature data to be input into the target business model. The multimodal feature data includes data of multiple business modalities, specifically, it can include data of image feature modalities corresponding to business feature data and target standard feature data, and data of text modalities corresponding to target image quality data. The multimodal feature data is obtained by splicing the feature data of these two business modalities, so it is called multimodal feature data.

[0078] In one embodiment, the target standard image includes M standard images, where M is a positive integer; the target standard feature data includes standard feature data corresponding to the M standard images respectively; the target image quality data includes image clarity data corresponding to the M standard images respectively; then, splicing the business feature data, the target standard feature data and the target image quality data to obtain multimodal feature data for inputting the target business model may include the following steps: ① performing a first splicing process on the standard feature data corresponding to each of the M standard images to obtain first splicing feature data; ② performing a second splicing process on the image clarity data corresponding to each of the M standard images to obtain second splicing feature data; the business modality indicated by the second splicing feature data is different from the business modality indicated by the first splicing feature data; ③ obtaining a splicing function for performing multimodal splicing processing, and splicing the business feature data, the first splicing feature data and the second splicing feature data through the splicing function to obtain multimodal feature data for inputting the target business model.

[0079] The number of the target standard images is M, where M is a positive integer, that is, the target standard image may include one or more standard images. For example, the target standard image may include five standard images in the standard image set with the greatest image similarity to the business image, such as target standard image I top5 ={I1,I2,I3,I4,I5}.

[0080] Correspondingly, the target standard feature data includes standard feature data corresponding to the M standard images respectively; the target image quality data includes image quality data corresponding to the M standard images respectively. Specifically, when the image quality is clarity, the image quality data corresponding to each standard image may be image clarity data, that is, the target image quality data includes image clarity data corresponding to the M standard images respectively.

[0081] The first splicing feature data may refer to feature data obtained by splicing the standard feature data corresponding to each standard image in the M standard images. The second splicing feature data may refer to feature data obtained by splicing the image clarity data corresponding to each standard image in the M standard images. It can be understood that the business modality indicated by the second splicing feature data is different from the business modality indicated by the first splicing feature data. Specifically, the business modality indicated by the first splicing feature data may be an image feature modality, and the business modality indicated by the second splicing feature data may be a text modality.

[0082] The concatenation function may refer to a function for performing multimodal concatenation processing, for example, the concatenation function may be a concat() function. Based on the concatenation function, the business feature data, the first concatenation feature data, and the second concatenation feature data may be concatenated to obtain multimodal feature data for inputting into the target business model.

[0083] For example, the process of determining multimodal feature data can be expressed by the following formula (2).

[0084] P = concat(F t ,F topM ,Q topM )Formula (2)

[0085] Among them, P represents multimodal feature data, concat() represents the concatenation function, and F t Represents the business feature data of the business image, F top5 Indicates the first splicing feature data corresponding to the standard feature data of M standard images (such as 5 standard images). top5 Represents second splicing feature data corresponding to standard feature data of M standard images (such as 5 standard images).

[0086] S104. When the model text prompt information for the target business model is obtained, the multimodal feature data and the model text prompt information are input into the target business model, and the target business model performs image quality detection on the multimodal feature data according to the model text prompt information to obtain image quality detection data.

[0087] The model text prompt information may be prompt information for prompting the target business model on a method for processing multimodal feature data. For example, the model text prompt information may include prompt instructions for telling the model what kind of task needs to be completed, and may also include output requirement prompt information for indicating the requirements of the result output by the target business model, or may also include other prompt information, which is not limited here.

[0088] When the multimodal feature data and the model text prompt information are input into the target business model, the multimodal model prompt information for the target business model can be generated based on the multimodal feature data and the model text prompt information, and then the multimodal model prompt information can be input into the target business model. The multimodal prompt model prompt information can refer to a prompt word (i.e., Prompt) for inputting the target business model, and Prompt is a technology based on artificial intelligence (AI) instructions, which outputs a language model through clear and specific guidance.

[0089] Among them, building multimodal model prompt information for the target business model based on multimodal feature data and model text prompt information can be generated based on prompt word engineering (i.e. Prompt Engineering). Through the prompt word engineering (i.e. Prompt Engineering), more accurate prompt words (i.e. multimodal model prompt information) can be generated, thereby obtaining more accurate image quality detection results.

[0090] Among them, the target business model may refer to a business model used for image quality detection. The target business model may be a large language model (i.e., an LLM model), and specifically may be a multimodal large language model (i.e., an MLLM model). The model structure in the target business model may be a transfomer structure (a network model structure), which may include an input layer, a codec, and an output layer. The input layer of the target business model may convert the input multimodal model prompt information into an embedding vector, and then the embedding vector may be encoded and decoded by the codec, thereby outputting the final result based on the output layer. Among them, a self-attention mechanism (Self-Attention) is used in the codec, and the use of the self-attention mechanism (Self-Attention) can capture the dependency between different business modalities. Thereby, the target business model can pay more attention to the correlation between the feature data of each modality, thereby improving the accuracy of the target business model in image quality detection.

[0091] The image quality detection data may be data obtained by performing quality detection on a business image. The image quality detection data may be expressed as an image quality score of the business image, or may be expressed as an image quality category determined based on the image quality score of the business image, or may be a quality qualification category determined based on the image quality score, and the quality qualification category may be used to indicate that the image quality is qualified or the image quality is unqualified. The image quality category may be a category obtained by dividing based on the image quality score, and specific reference may be made to the relevant description of the above table (1). The quality qualification category may be determined based on whether the image quality score is greater than or equal to a scoring threshold. If the image quality score of the business image is greater than or equal to the scoring threshold, the quality qualification category of the business image indicates that the image quality is qualified. If the image quality score of the business image is less than the scoring threshold, the quality qualification category of the business image indicates that the image quality is unqualified.

[0092] In one embodiment, multimodal feature data and model text prompt information are input into a target business model, and the target business model performs image quality detection on the multimodal feature data according to the model text prompt information to obtain image quality detection data, which may include the following steps: ① constructing multimodal model prompt information for the target business model based on the multimodal feature data and the model text prompt information, and inputting the multimodal model prompt information into the target business model; ② the target business model performs image quality detection on the multimodal feature data according to the multimodal model prompt information to obtain a first quality detection result of the business image; ③ determining the first quality detection result as the image quality detection data.

[0093] Among them, the introduction of constructing multimodal prompt information can refer to the above-mentioned related description, which will not be repeated here. The introduction of the model structure of the target business model can refer to the above-mentioned related description. For example, the multimodal prompt information can prompt the target business model to obtain the quality detection result of the image corresponding to the business feature data in the multimodal feature data based on the multimodal feature data detection, thereby outputting the image quality detection data.

[0094] Among them, the first quality detection result can be a quality detection result determined based on multimodal feature data. The first quality detection result can be expressed as an image quality score for the business image. Further, the first quality detection result can be determined as the final image quality detection data, for example, it can be determined as the image quality detection data based on the image quality score of the business image indicated in the first quality detection result; or, the image quality category to which the business image belongs can be determined based on the image quality score of the business image indicated in the first quality detection result, and the determined image quality category can be determined as the image quality detection data; or, the quality qualification category of the business image can be determined based on the image quality score of the business image indicated in the first quality detection result, and the determined quality qualification category can be determined as the image quality detection data.

[0095] Here, combined with the diagram, the image quality detection process of business images based on multimodal feature data is explained. Figure 5 , Figure 5 FIG. 1 is a flow chart of an image quality detection process provided by an embodiment of the present application. Figure 5 As shown, firstly, the business image under the target business scenario can be obtained (such as Figure 5 501a in ), so that the feature extractor (such as Figure 5 503a in FIG. 5 ) performs feature extraction processing on the business image to obtain business feature data (such as Figure 5 In addition, the standard feature data of each standard image in the standard image set corresponding to the target business scenario can be obtained (such as Figure 5 ), which may be a feature extractor (such as Figure 5 503a in FIG. 5 ) for a standard image set corresponding to the target business scenario (such as Figure 5 The feature extraction process is performed on each standard image in the image retrieval module (such as 502a in FIG. 1 ). Figure 5 506a in FIG) determines a target standard image (such as a target standard image matching the business image) based on the standard feature data and business feature data of each standard image in the standard image set corresponding to the target business scene. Figure 5 The specific image retrieval process can refer to the relevant description of step S102, which will not be described here. The target standard image may include multiple standard images, such as image Itop1, image Itop2, and image ItopM. Then, the target image quality data corresponding to the target standard image can be obtained (such as Figure 5508a), the target image quality data includes image quality data corresponding to each standard image in the target standard image, for example, quality data Qtop1 corresponding to image Itop1, quality data Qtop2 corresponding to image Itop2, quality data QtopM corresponding to image ItopM, and so on.

[0096] Furthermore, the target standard feature data corresponding to the target standard image can be obtained (such as Figure 5 509a), the target standard feature data may include standard feature data corresponding to each standard image in the target standard image, for example, feature data Ftop1 corresponding to image Itop1, feature data Ftop2 corresponding to image Itop2, feature data FtopM corresponding to image ItopM, and so on.

[0097] Furthermore, multimodal feature data (such as Figure 5 The specific construction method can refer to the relevant description of the above step S103, which will not be repeated here.

[0098] Further, the prompt word engineering (such as Figure 5 511a in FIG) determines the multimodal model prompt information based on the multimodal feature data and the model text prompt information, so that the target business model (such as Figure 5 512a in FIG. 5 ) performs image quality detection on the multimodal feature data according to the multimodal model prompt information to obtain image quality detection data of the business image (such as Figure 5 513a in FIG. 1 ).

[0099] It is understandable that in the embodiment of the present application, a multimodal retrieval enhanced generation (RAG) method is adopted (i.e., a target standard image matching the business image is retrieved), and image retrieval and MLLM technology are comprehensively utilized. By combining the business image with the standard image set provided by the business party corresponding to the target business scene, the top 5 reference images (i.e., M standard images) closest to the semantics and clarity of the business image can be retrieved, and then multimodal feature data are formed, and then the multimodal feature data is handed over to the MLLM model (i.e., the target business model) for comprehensive judgment. This image retrieval process not only improves the accuracy of clarity judgment, but also enhances the system's adaptability to different business scenarios. In addition, the standard image set corresponding to each business scene can be replaced by the relevant management personnel of the business party at any time, so that the image quality judgment standard of each business scene can be adjusted more conveniently, thereby improving the flexibility of image quality detection.

[0100] Optionally, after obtaining the first quality detection result, the first quality detection result may be adjusted based on the image quality rule associated with the target business scenario to obtain a second quality detection result, and then the final image quality detection data is determined based on the first quality detection result and the second quality detection result. The specific process can be referred to as follows Figure 5 Description of the illustrated embodiment.

[0101] For further information, see Figure 6 , Figure 6 is a flowchart of an image processing method provided by an embodiment of the present application. The method can be executed by a computer device, for example, the computer device can be the above-mentioned Figure 1 The method may at least include the following steps S201 to S208.

[0102] S201, obtaining a business image to be subjected to image quality detection; the business image is an image in a target business scenario; the target business scenario refers to a scene in at least one business scenario; and one business scenario corresponds to one standard image set.

[0103] S202: Retrieve a target standard image that matches the business image from a standard image set corresponding to the target business scenario, and obtain target image quality data bound to the target standard image.

[0104] S203. When the business feature data of the business image and the target standard feature data of the target standard image are obtained, the business feature data, the target standard feature data and the target image quality data are spliced ​​to obtain multimodal feature data for inputting into the target business model.

[0105] Among them, the introduction of steps S201-S203 can refer to the relevant description of the above steps S101-S103, which will not be repeated here.

[0106] Based on the above description, it can be known that the business feature data is obtained by performing feature extraction processing on the business image through a feature extraction processor, and the feature extraction processor includes a semantic feature extractor and a quality feature extractor.

[0107] In one embodiment, the business feature data is obtained by performing feature extraction processing on the business image through a feature extraction processor, and the feature extraction processor includes a semantic feature extractor and a quality feature extractor; then, the method for determining the business feature data of the business image may include the following steps: ① Performing semantic feature extraction processing on the business image through the semantic feature extractor to obtain the semantic feature data of the business image; ② Performing quality feature extraction processing on the business image through the quality feature extractor to obtain the quality feature data of the business image; ③ Performing feature fusion processing on the semantic feature data and the quality feature data to obtain the business feature data of the business image.

[0108] Among them, the semantic feature extractor can be an extractor for extracting the semantic features (i.e., semantic feature data) of an image (such as a business image). The quality feature extractor can be an extractor for extracting the features (i.e., quality feature data) of an image (such as a business image) in terms of image quality. Among them, semantic feature data refers to the semantic features of an image, which are used to characterize information about the content of the image. Quality feature data refers to the features of an image (such as a business image) in terms of image quality, which are used to characterize information about the quality of the image. In the present application, the network structures of the semantic feature extractor and the quality feature extractor can be the same or different, and are not limited here. For example, the network structure of the semantic feature extractor and the quality feature extractor can be a CNN (convolutional neural network) network.

[0109] The feature fusion processing of the semantic feature data and the quality feature data may be performed by fusing the semantic feature data and the quality feature data through a feature fusion network. The feature fusion network may be used for a network that fuses the semantic feature data and the quality feature data. For example, the feature fusion network may be a transfomer (a network structure), which is not limited here.

[0110] For example, see Figure 7 , Figure 7 FIG. 1 is a flow chart of a feature extraction process provided by an embodiment of the present application. Figure 7 As shown, after obtaining the business image (such as Figure 7 ), the semantic feature extractor (such as Figure 7 702a in FIG. 702a ) extracts semantic features from the business image to obtain semantic feature data, and can be extracted by a quality feature extractor (such as Figure 7 703a in FIG. 7A ) extracts quality features from the business image to obtain quality feature data. Then, the feature fusion network (such as Figure 7 704a in FIG. 7 ) performs feature fusion on the semantic feature data and the quality feature data to obtain the business feature data of the business image (such as Figure 7705a in FIG. 1 ).

[0111] It can be understood that both the semantic feature extractor and the quality feature extractor can be obtained after training based on corresponding training data.

[0112] In one embodiment, the method for determining a semantic feature extractor may include the following steps: ① obtaining a first sample image for training an initial semantic feature extractor; ② performing semantic feature extraction processing on the first sample image by the initial semantic feature extractor to obtain sample semantic feature data of the first sample image; ③ performing image scaling processing on the first sample image to obtain a scaled sample image corresponding to the first sample image; ④ performing semantic feature extraction processing on the scaled sample image by the initial semantic feature extractor to obtain scaled semantic feature data of the scaled sample image; ⑤ adjusting parameters of the initial semantic feature extractor based on the sample semantic feature data and the scaled semantic feature data, and determining the initial semantic feature extractor after the parameter adjustment as the semantic feature extractor.

[0113] The initial semantic feature extractor may be an untrained semantic feature extractor, and the network structure of the initial semantic feature extractor is consistent with the network structure of the target semantic feature extractor. The first sample image may be a sample image used to train the initial semantic feature extractor.

[0114] The sample semantic feature data may refer to feature data obtained by performing semantic feature extraction processing on the first sample image by the initial semantic feature extractor, which may characterize the semantic features of the first sample image. The method for determining the sample semantic feature data may refer to the introduction of determining the semantic feature data of the business image above, which will not be described in detail here.

[0115] The scaled sample image may be an image obtained by scaling the first sample image. The scaled sample image is consistent with the first sample image in image content, but has a different image size, which means that the image quality is different.

[0116] The scaled semantic feature data may refer to feature data obtained by performing semantic feature extraction processing on the scaled sample image by the initial semantic feature extractor. The method for determining the scaled semantic feature data may refer to the above description of determining the semantic feature data of the business image, which will not be described in detail here.

[0117] It can be understood that the scaled sample image is consistent with the first sample image in image content. Therefore, the sample semantic feature data and the scaled semantic feature data extracted by the same initial semantic feature extractor should be as consistent as possible. Therefore, the loss information for the initial semantic feature extractor (such as which can be recorded as the first loss information) can be calculated based on the sample semantic feature data and the scaled semantic feature data, and then the network parameters of the initial semantic feature extractor can be adjusted by the first loss information until the first loss information converges, so that the trained initial semantic feature extractor can extract the semantic features of the image, and then the initial semantic feature extractor after parameter adjustment can be determined as the semantic feature extractor.

[0118] For example, see Figure 8 , Figure 8 FIG. 1 is a flow chart of a training process of a semantic feature extractor provided in an embodiment of the present application. Figure 8 As shown, firstly, a first sample image 801a for training an initial semantic feature extractor can be obtained, and the first sample image 801a can be scaled to obtain a scaled sample image 802a. It can be seen that the first sample image 801a and the scaled sample image 802a have the same image content but different image sizes. Then, the initial semantic feature extractor (such as Figure 8 803a in FIG. 8 ) performs semantic feature extraction processing on the first sample image to obtain sample semantic feature data of the first sample image (such as Figure 8 804a in FIG), through an initial semantic feature extractor (such as Figure 8 803a in FIG. 8 ) performs semantic feature extraction processing on the scaled sample image to obtain scaled semantic feature data of the scaled sample image (such as Figure 8 805a in FIG), and then the first loss information (such as Figure 8 As shown in 806a in FIG, the network parameters of the initial semantic feature extractor are adjusted based on the first loss information until the loss converges, so that the initial semantic feature extractor after the network parameter adjustment can be determined as the semantic feature extractor.

[0119] In one embodiment, a method for determining a quality feature extractor may include the following steps: ① obtaining a second sample image for training an initial quality feature extractor; ② performing quality feature extraction processing on the second sample image by the initial quality feature extractor to obtain sample quality feature data of the second sample image; ③ performing image segmentation on the second sample image to obtain multiple image slices corresponding to the second sample image, and randomly combining the multiple image slices to obtain a combined sample image corresponding to the second sample image; an arrangement and display order of the multiple image slices in the combined sample image is different from an arrangement and display order of the multiple image slices in the second sample image; ④ performing quality feature extraction processing on the combined sample image by the initial quality feature extractor to obtain combined quality feature data of the combined sample image; ⑤ adjusting parameters of the initial quality feature extractor based on the sample quality feature data and the combined quality feature data, and determining the initial quality feature extractor after the parameter adjustment as the quality feature extractor.

[0120] The initial quality feature extractor may be an untrained quality feature extractor, and the network structure of the initial quality feature extractor is consistent with the network structure of the target quality feature extractor. The second sample image may be a sample image used to train the initial quality feature extractor.

[0121] The sample quality feature data may refer to feature data obtained by performing quality feature extraction processing on the second sample image by the initial quality feature extractor, which may characterize the quality features of the second sample image. The method for determining the sample quality feature data may refer to the introduction of determining the quality feature data of the business image above, which will not be described in detail here.

[0122] Among them, the combined sample image can be an image obtained by randomly combining multiple image slices of the second sample image. The arrangement and display order of the multiple image slices in the combined sample image is different from the arrangement and display order of the multiple image slices in the second sample image, so that the combined sample image is consistent with the second sample image in terms of image quality, but the image content between the two is disrupted, that is, the image content is different. Among them, the image slice refers to the slice obtained by image segmentation of the second sample image. For example, the size of each image slice obtained by segmentation can be the same or different, which is not limited here. The present application does not limit the number of image slices obtained by segmenting the second sample image, which can be determined according to actual needs. For example, the image segmentation of the second sample image can be to divide the second sample image into four equal parts to obtain four image slices of the same size. For example, the second sample image can be divided into six equal parts to obtain six image slices of the same size.

[0123] It can be understood that the combined sample image and the second sample image are consistent in image quality. Therefore, the sample quality feature data and the combined quality feature data extracted by the same initial quality feature extractor should be as consistent as possible. Therefore, the loss information for the initial quality feature extractor (such as which can be recorded as the second loss information) can be calculated based on the sample quality feature data and the combined quality feature data, and then the network parameters of the initial quality feature extractor can be adjusted by the second loss information until the second loss information converges, so that the trained initial quality feature extractor can extract the quality characteristics of the image, and then the initial quality feature extractor after parameter adjustment can be determined as the quality feature extractor.

[0124] For example, see Fig. 9 , Fig. 9 FIG. 1 is a flow chart of a training process of a quality feature extractor provided in an embodiment of the present application. Fig. 9 As shown, firstly, a second sample image 901a for training an initial quality feature extractor can be obtained, and then the second sample image is segmented to obtain a plurality of image slices corresponding to the second sample image (such as Fig. 9 901b in the figure), wherein the multiple image slices may include image slice ①, image slice ②, ... image slice ⑥, etc. Then, the multiple image slices may be sliced ​​and combined (such as randomly combined) to obtain a combined sample image 902a. It can be seen that in the combined sample image 902a, the arrangement and display order of each image slice is different from the arrangement and display order of multiple image slices in the second sample image 901a. Obviously, the second sample image 901a and the combined sample image 902a have different image contents and different image sizes, so that the image quality between the two is consistent. Then, the initial quality feature extractor (such as Fig. 9 903a in FIG. 9 ) performs quality feature extraction processing on the second sample image to obtain sample quality feature data of the second sample image (such as Fig. 9 904a in FIG), through an initial quality feature extractor (such as Fig. 9 903a in FIG. 9 ) performs quality feature extraction processing on the combined sample image to obtain combined quality feature data of the combined sample image (such as Fig. 9 905a in FIG), and then the second loss information (such as Fig. 9 As shown in 906a in FIG. 1 , the network parameters of the initial quality feature extractor are adjusted based on the second loss information until the loss converges, so that the initial quality feature extractor after the network parameter adjustment can be determined as the quality feature extractor.

[0125] S204: When the model text prompt information for the target business model is obtained, the image quality rule under the target business scenario is obtained; the image quality rule is configured for the target business scenario.

[0126] Among them, the image quality rule may refer to a rule for determining image quality. The image quality rule under the target business scenario may be configured for the target business scenario, for example, it may be configured by the scene management object (such as management user) of the corresponding business scenario. It is understandable that in the embodiment of the present application, a corresponding image quality rule may be configured for each business scenario in the business scenario set. Therefore, when the target business scenario is determined, the image quality rule under the target business scenario may be determined based on the target business scenario.

[0127] Among them, the image quality rule may include N sub-rules, N is a positive integer; each sub-rule may be a judgment logic in the target business scenario. For example, in the case where the business scenario is a resource recommendation scenario, an additional image quality rule may be configured so that when the image type of the business image is the target type, a preset quality score may be added to the first quality detection result. Even if the first quality detection result indicates that the business image is blurry, the image quality score detected by the image quality rule may be improved, thereby avoiding reducing the recommendation probability of the business image of the target type. The target type may be a specified type among a plurality of image types that classify the content of the business image. For example, in some business scenarios, the target type may be an image type that is easy to blur. For example, the target type may be a person selfie type, a night scene type, a KTV scene type, a sports scene type, etc., which are not limited here. The image type to which the business image belongs may be obtained by classifying the business feature data of the business image through the target business model, or may be obtained by classification and identification through other image classification models, or may be annotated by the business pusher (such as a user) when publishing or uploading the business image, which are not limited here. Based on this, in this application, some special types of images can be specially processed more flexibly through the configuration of image quality rules. In addition, the configuration of the image quality rules can be customized at any time without affecting the processing logic of the target business model, so it has greater flexibility, so that image quality detection can be more flexibly performed on images in various business scenarios.

[0128] S205: construct multimodal model prompt information for the target business model based on the image quality rule, the multimodal feature data and the model text prompt information, and input the multimodal model prompt information into the target business model.

[0129] Among them, the introduction of the multimodal model prompt information can refer to the above-mentioned related description, which will not be repeated here. For the sake of distinction, the multimodal model prompt information constructed based on the multimodal feature data and the model text prompt information can be referred to as the first multimodal model prompt information, and the multimodal model prompt information constructed based on the image quality rules, the multimodal feature data and the model text prompt information can be referred to as the second multimodal model prompt information. Furthermore, when the second multimodal model prompt information is constructed, the second multimodal model prompt information can be input into the target business model. For example, the second multimodal prompt information can prompt the target business model to determine the first quality detection result based on the multimodal feature data, and adjust the quality result of the first quality detection result through the image quality rule to obtain the second quality detection result, thereby outputting the image quality detection data determined by the first quality detection result and the second quality detection result.

[0130] S206: The target business model performs image quality detection on the multimodal feature data according to the multimodal model prompt information to obtain a first quality detection result of the business image.

[0131] Among them, the method for determining the first quality detection result can refer to the above related description, which will not be repeated here.

[0132] S207: Obtain a second quality detection result for the business image through the image quality rule and the first quality detection result.

[0133] The second quality detection result may be a quality detection result determined based on an image quality rule, and specifically may refer to a result obtained by adjusting the quality result of the first quality detection result based on the image quality rule. The second quality detection result may be expressed as an image quality score.

[0134] In one embodiment, the image quality rule includes N sub-rules, where N is a positive integer; then, obtaining a second quality detection result for the business image through the image quality rule and the first quality detection result may include the following steps: ① determining a target sub-rule that matches the business image from the N sub-rules; ② calling the target sub-rule to adjust the quality result of the first quality detection result to obtain a second quality detection result for the business image.

[0135] The target sub-rule may be a sub-rule that matches the business image. For example, the N sub-rules may include sub-rule R1, sub-rule R2 and sub-rule R3, wherein sub-rule R1 is: when the image type of the business image is a selfie type, the image quality score indicated by the first quality detection result is increased by a first preset quality score; sub-rule R2 is: when the image type of the business image is a motion scene type, the image quality score indicated by the first quality detection result is increased by a second preset quality score; sub-rule R3 is: when the image type of the business image is a night scene type, the image quality score indicated by the first quality detection result is increased by a third preset quality score. Then, determining the target sub-rule that matches the business image from N sub-rules can be: obtaining the image type corresponding to the business image, and determining the sub-rule that matches the business image based on the image type of the business image. For example, when the image type of the business image is a selfie type, sub-rule R1 can be determined as the target sub-rule, and then the target sub-rule is called to adjust the quality result of the first quality detection result. The image quality score indicated by the first quality detection result can be increased by a first preset quality score according to the indication of the target sub-rule (i.e., sub-rule R1) to obtain a second quality detection result.

[0136] For example, any sub-rule can be expressed by the following formula (3):

[0137] R j :If K t = "K j ", then Q t2 =E(Q t1 )Formula (3)

[0138] Among them, R j represents the jth sub-rule in the image quality rule, K t Indicates the image type of the business image. K1 indicates image type K j , Q t2 is the second quality detection result of the business image, Q t1 is the first quality detection result of the business image, and E() represents a function for adjusting the first quality detection result.

[0139] Furthermore, calling the target sub-rule to adjust the quality result of the first quality detection result can be expressed by the following formula (4):

[0140] Q t2 =override(Q t1 , R j )Formula (4)

[0141] Among them, Q t2Indicates the second quality detection result of the business image, Q t1 Represents the first quality detection result of the business image, R j Indicates the target sub-rule that matches the business image, override() means calling the target sub-rule R j The first quality test result Q t1 Function to make quality result adjustments.

[0142] S208: Determine image quality detection data based on the first quality detection result and the second quality detection result.

[0143] It can be understood that the image quality detection data is determined based on the first quality detection result and the second quality detection result. The result can be obtained by weighted fusion of the first quality detection result and the second quality detection result. Specifically, the image quality score indicated by the first quality detection result and the image quality score indicated by the second quality detection result can be weighted averaged to obtain the final target image quality score, thereby determining the final image quality detection data based on the target image quality score. For example, the target image quality score can be determined as the image quality detection data, or the final image quality category or quality acceptance category can be determined as the image quality detection data based on the target image quality score.

[0144] It is understandable that in the embodiment of the present application, through the mechanism of image quality rules, the entire image quality detection system has a high level of flexibility and customizability, and the business parties corresponding to each business scenario can flexibly adjust the image quality rules of the corresponding business scenario. In addition, when performing image quality detection on the business image under the target business scenario, the specific business rules (i.e., the image quality rules corresponding to the target business scenario) can be directly input into the target business model, and the target business model will comprehensively consider these rules to achieve intelligent judgment when generating the discrimination result. In addition, the target business model can combine the retrieved target standard image and its clarity score (i.e., image quality data) to comprehensively improve the reliability of the discrimination of image quality detection. It is understandable that the design of the entire image quality detection system supports rapid adaptation to new clarity audit standards or modifications to existing standards, that is, the standard image set and image quality rules corresponding to each business scenario can be adjusted. This feature enables the entire image quality detection system to continuously meet the changing needs of the business party and achieve efficient clarity audit (i.e., image quality audit).

[0145] In one embodiment, the first quality detection result is used to indicate a first image quality score of a business image, and the second quality detection result is used to indicate a second image quality score of the business image; then, determining the image quality detection data based on the first quality detection result and the second quality detection result may include the following steps: ① obtaining a first scoring weight for the first image quality score, and a second scoring weight for the second image quality score; ② based on the first scoring weight and the second scoring weight, weightedly fusing the first image quality score and the second image quality score to obtain the image quality detection data.

[0146] Among them, the first image quality score may refer to the image quality score indicated by the first quality detection result, and the second image quality score may refer to the image quality score indicated by the second quality detection result. The introduction of the image quality score can refer to the above-mentioned related description and will not be repeated here.

[0147] Among them, the first scoring weight may refer to the weight corresponding to the first image quality score, and the second scoring weight may refer to the weight corresponding to the second image quality score. It can be understood that the first scoring weight and the second scoring weight can characterize the importance of the first image quality score and the second image quality score when determining the final image quality detection data. The larger the scoring weight, the greater the impact of the corresponding image quality score (such as the first image quality score or the second image quality score) on the determined image quality detection data. It can be understood that the first scoring weight and the second scoring weight can be preset according to actual business needs, and are not limited here.

[0148] Among them, the first image quality score and the second image quality score are weightedly fused to obtain the image quality detection data, and the first image quality score and the second image quality score are weighted averaged to obtain the target image quality score, and then the image quality detection data is determined based on the target image quality score.

[0149] For example, the image quality detection data of the business image may be determined by referring to the following formula (5).

[0150] Q t =αQ t1 +βQ t2 Formula (5)

[0151] Among them, Q t represents the target image quality score indicated by the image quality detection data of the business image, Q t1 represents the first image quality score, Q t2 represents the second image quality score. Correspondingly, α represents the first image quality score Q t1 The corresponding first score weight, β represents the second image quality score Q t2The corresponding second scoring weight.

[0152] It can be understood that in the embodiment of the present application, the target business model can be obtained by fine-tuning a large language model obtained by pre-training with a large amount of sample data.

[0153] Here, combined with diagrams, the image quality detection process of business images based on multimodal feature data and image quality rules is explained. For example, see Fig.10 , Fig.10 FIG. 1 is a flow chart of another image quality detection process provided by an embodiment of the present application. Fig.10 As shown, after obtaining the business image under the target business scenario (such as Fig.10 After 1001a), we can Fig.10 The steps 1001a-1010a in FIG. 1 are used to determine the multimodal feature data (eg Fig.10 1010a), wherein the multimodal feature data (such as Fig.10 The determination method of 1010a) in can refer to the above Figure 5 The relevant descriptions of 501a-510a in are not repeated here. Further, the image quality rules corresponding to the target business scenario can be obtained (such as Fig.10 1011a), by prompt word engineering (such as Fig.10 1012a in FIG. 1012a ) determines the multimodal model prompt information based on the image quality rule, the multimodal feature data and the model text prompt information, so that the target business model (such as Fig.10 1013a in FIG. 1013a ) performs image quality detection on the multimodal feature data according to the multimodal model prompt information to obtain image quality detection data of the business image (such as Fig.10 The specific process of determining the image quality detection data can refer to the above related description, which will not be repeated here.

[0154] In one embodiment, the embodiment of the present application may further include the following steps: ① obtaining sample training data for an initial business model; the sample training data includes a third sample image, an image quality label associated with the third sample image, and a sample standard image associated with the third sample image; the sample standard image is bound with sample image quality data; ② when the sample feature data corresponding to the third sample image and the sample standard feature data corresponding to the sample standard image are obtained, the sample feature data, the sample standard feature data and the sample image quality data are spliced ​​to obtain sample multimodal feature data for inputting the initial business model; ③ image quality detection is performed on the sample multimodal feature data through the initial business model to obtain a predicted image quality detection result; ④ based on the predicted image quality detection result and the image quality label, the model parameters of the initial business model are fine-tuned to obtain a target business model.

[0155] The initial business model may refer to a business model that has not been fine-tuned. For example, the initial business model may be a large language model obtained through training with a large amount of corpus. The model structure of the initial business model may refer to the introduction of the model structure of the target business model, which will not be described here.

[0156] The sample training data may be sample data used to train the initial business model. The third sample image in the sample training data may refer to a sample image used for image quality detection, and the image quality label may be a label of image quality annotated on the third sample image, used to characterize the actual image quality of the third sample image. The image quality label may be expressed as an image quality score, image quality category, or quality qualification category (i.e., image quality is qualified or unqualified) of the third sample image, which is not limited here.

[0157] The sample standard image associated with the third sample image may refer to a standard image corresponding to the third sample image, and the sample standard image may be used as a reference for image quality detection of the third sample image. The sample image quality data may refer to the image quality data bound to the sample standard image, and the introduction of the sample image quality data may refer to the relevant description of the image quality data of the above-mentioned standard image, which will not be repeated here.

[0158] The sample feature data may refer to feature data corresponding to the third sample image, which may be obtained by extracting features from the third sample image using the feature extractor. The method for determining the sample feature data may refer to the relevant description of the business feature data of the business image, which will not be described in detail here. The sample standard feature data may refer to feature data corresponding to the sample standard image, which may be obtained by extracting features from the sample standard image using the feature extractor. The method for determining the sample feature data may refer to the relevant description of the standard feature data of the standard image, which will not be described in detail here.

[0159] The sample multimodal feature data is feature data obtained by splicing the sample feature data, sample standard feature data and sample image quality data. The specific acquisition process can be found in the relevant description of the above multimodal feature data, which will not be repeated here.

[0160] The predicted image quality detection result may refer to the result obtained by the initial business model performing image quality detection on the sample multimodal feature data. For details, please refer to the relevant description of determining the above-mentioned image quality detection data, which will not be repeated here. The predicted image quality detection result may be expressed as an image quality score, a probability for each image quality category, or a probability of qualified image quality, which is not limited here.

[0161] Furthermore, the third loss information can be determined based on the predicted image quality detection result and the image quality label, and the model parameters of the initial business model can be adjusted based on the third loss information until the loss converges, so that the predicted image quality detection result obtained by the initial business model after the parameter adjustment can be closer to the image quality label, and the initial business model after the parameter adjustment can be determined as the target business model. The target business model has the ability to perform image quality detection on images based on multimodal feature data, and the target business model can be used to perform image quality detection on business images to obtain image quality detection data suitable for the target business scenario.

[0162] In one embodiment, each business scene in at least one business scene has a corresponding scene identifier; obtaining a business image to be subjected to image quality detection includes: ① obtaining a business video to be subjected to image quality detection, and selecting a target video frame from multiple video frames included in the business video; the business video is associated with a target scene identifier; ② searching for a business scene that matches the target scene identifier from at least one business scene; ③ if a target business scene that matches the target scene identifier is found, determining the target video frame as the business image to be subjected to image quality detection.

[0163] The introduction of at least one business scenario can refer to the above-mentioned related description, which will not be repeated here. The scenario identifier can be a unique identifier corresponding to a business scenario, and each business scenario in the business scenario set has a corresponding scenario identifier, so when the scenario identifier is obtained, the corresponding business scenario can be determined from the business scenario set based on the scenario identifier.

[0164] The business video may refer to a video to be subjected to image quality detection, and the business video may be a video published or uploaded by a business object (such as a user). The business video may include multiple video frames, which are the basic components of a video, and each video frame is a still picture.

[0165] The target video frame may be a video frame selected from a plurality of video frames included in the business video for image quality detection. Selecting the target video frame from a plurality of video frames may be a random selection of video frames from a plurality of video frames, or may be a selection of video frames according to certain video frame sampling rules, which are not limited here. The video frame sampling rule may refer to a rule for selecting video frames from a plurality of video frames in a business video. For example, the video frame sampling rule may include: determining the number of sampling frames according to the video length of the business video, and selecting video frames from the business video frames as business images based on the number of sampling frames; or, the video frame sampling rule may include: determining key video frames from a plurality of video frames in the business video frames, and selecting video frames from the key video frames as business images. The key video frame may refer to a key frame in a business video, and the key frame may refer to the frame where the key action is located in the motion change of a character or object in animation or video production. Alternatively, the video sampling rule may also be some other rules, which are not limited here.

[0166] The business video may be associated with a target scene identifier, which refers to a scene identifier corresponding to the business scene to which the business video belongs. Furthermore, based on the target scene identifier, a business scene matching the target scene identifier may be searched from multiple business scenes included in the business scene set, so that the business scene matching the target scene identifier may be determined as the target business scene. It is understandable that if a target business scene matching the scene identifier is found, it means that the business scene to which the business video belongs belongs to the business scene set, and then image quality detection can be performed through the embodiment of the present application. Conversely, if the target business scene matching the scene identifier is not found, it means that the business scene to which the business video belongs does not belong to the business scene set, and it is difficult to subsequently obtain the relevant standard image set corresponding to the business scene to which the business video belongs, and it is not necessary to perform image quality detection on it through the embodiment of the present application.

[0167] It can be understood that the number of target video frames selected from multiple video frames in the business video can be one or more, each selected target video frame can be used as a business image, and the business scene to which each business image belongs is the target business scene, so that the image quality detection can be performed on one or more target video frames selected from the business video frame. Then, the video quality detection result of the business video can be determined based on the image quality detection result corresponding to each target video frame. The video quality detection result may refer to the detection result of the video quality of the business video. Optionally, determining the video quality detection result of the business video may be to calculate the average video frame score of the image quality score indicated by the image quality detection result corresponding to each target video frame (i.e., the average value of the image quality scores corresponding to each target video frame), thereby determining the video quality detection result based on the average video frame score, for example, when the average video frame score is greater than or equal to a certain video score threshold, the video quality detection result indicating that the image quality of the business video is qualified is determined, and vice versa, when the average video frame score is less than a certain video score threshold, the video quality detection result indicating that the image quality of the business video is unqualified is determined. Furthermore, corresponding business processing can be performed on the business video in the corresponding target business scenario based on the video quality detection result. For example, in a resource recommendation scenario, if the video quality detection result of the business video in the video resource indicates that the video quality of the business video is unqualified, the probability of recommending the video resource corresponding to the business video can be reduced.

[0168] It is understandable that the image processing method provided in the embodiment of the present application can be encapsulated as a reusable image quality detection component for wide use in multiple business scenarios. For example, the image quality detection component can adopt a preset protocol, such as inferno (a type of protocol) reasoning framework protocol, and its input parameters can include parameters such as the video identification of the business video (or the image identification of the business image), the video source (i.e., the business scene identification), and the title. The image quality detection component can be quickly integrated into different business scenarios through efficient interface design, meet various content quality assessment requirements, and help business parties improve efficiency and user experience in multiple fields such as video processing, content recommendation, and creation optimization. For example, the component can adopt a preset protocol (such as inferno reasoning framework protocol), and its input parameters can include parameters such as the video identification of the business video, the video source (i.e., the business scene identification), and the title. Thus, in the actual business processing process, the image quality detection can be quickly performed by calling the image quality detection component to improve the efficiency of image quality detection.

[0169] See also Fig.11 , Fig.11 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig.11As shown, the image processing device 1 may be a computer program (including program code) running on a computer device (e.g., the server 100a mentioned above), for example, the image processing device 1 is an application software; it can be understood that the image processing device 1 can be used to execute the corresponding steps of the image processing method provided in the embodiment of the present application. Fig.11 As shown, the image processing device 1 may include: an image acquisition unit 11, an image retrieval unit 12, a feature data splicing unit 13, and an image quality detection unit 14;

[0170] An image acquisition unit 11 is used to acquire a business image to be subjected to image quality detection; the business image is an image in a target business scenario; the target business scenario refers to a scene in at least one business scenario; one business scenario corresponds to one standard image set;

[0171] An image retrieval unit 12 is used to retrieve a target standard image that matches the business image from a standard image set corresponding to a target business scenario, and obtain target image quality data bound to the target standard image;

[0172] The feature data splicing unit 13 is used to splice the business feature data, the target standard feature data and the target image quality data to obtain multimodal feature data for inputting the target business model when the business feature data of the business image and the target standard feature data of the target standard image are acquired;

[0173] The image quality detection unit 14 is used to input the multimodal feature data and the model text prompt information into the target business model when the model text prompt information for the target business model is obtained, and the target business model performs image quality detection on the multimodal feature data according to the model text prompt information to obtain image quality detection data.

[0174] In one embodiment, the image retrieval unit 12 is specifically configured to:

[0175] Performing feature extraction processing on the business image through a feature extractor to obtain business feature data of the business image;

[0176] Obtain standard feature data corresponding to each standard image in the standard image set corresponding to the target business scenario; the standard feature data corresponding to any standard image is obtained by performing feature extraction processing on any standard image by a feature extractor;

[0177] Determining the image similarity between each standard image and the business image based on the feature similarity between the business feature data and the standard feature data corresponding to each standard image;

[0178] From the standard image set corresponding to the target business scene, a standard image whose image similarity with the business image satisfies a similarity condition is searched, and the searched standard image is determined as a target standard image matching the business image.

[0179] In one embodiment, the target standard image includes M standard images, where M is a positive integer; the target standard feature data includes standard feature data corresponding to the M standard images respectively; and the target image quality data includes image definition data corresponding to the M standard images respectively;

[0180] The feature data splicing unit 13 is specifically used for:

[0181] Performing a first splicing process on the standard feature data corresponding to each standard image in the M standard images to obtain first splicing feature data;

[0182] Performing a second splicing process on the image definition data corresponding to each of the M standard images to obtain second splicing feature data; the business modality indicated by the second splicing feature data is different from the business modality indicated by the first splicing feature data;

[0183] A splicing function for performing multimodal splicing processing is obtained, and the business feature data, the first splicing feature data and the second splicing feature data are spliced ​​through the splicing function to obtain multimodal feature data for inputting into a target business model.

[0184] In one embodiment, the image quality detection unit 14 is specifically configured to:

[0185] Constructing multimodal model prompt information for a target business model based on the multimodal feature data and the model text prompt information, and inputting the multimodal model prompt information into the target business model;

[0186] The target business model performs image quality detection on the multimodal feature data according to the multimodal model prompt information to obtain a first quality detection result of the business image;

[0187] The first quality detection result is determined as image quality detection data.

[0188] In one embodiment, the image quality detection unit 14 is specifically configured to:

[0189] Obtain image quality rules under the target business scenario; the image quality rules are configured for the target business scenario;

[0190] Building multimodal model prompt information for a target business model based on image quality rules, multimodal feature data, and model text prompt information, and inputting the multimodal model prompt information into the target business model;

[0191] The target business model performs image quality detection on the multimodal feature data according to the multimodal model prompt information to obtain a first quality detection result of the business image;

[0192] Obtaining a second quality detection result for the business image through the image quality rule and the first quality detection result;

[0193] Image quality detection data is determined based on the first quality detection result and the second quality detection result.

[0194] In one embodiment, the image quality rule includes N sub-rules, where N is a positive integer;

[0195] The image quality detection unit 14 is specifically used for:

[0196] Determine a target sub-rule matching the business image from the N sub-rules;

[0197] The target sub-rule is called to adjust the quality result of the first quality detection result to obtain a second quality detection result for the business image.

[0198] In one embodiment, the first quality detection result is used to indicate a first image quality score of the business image, and the second quality detection result is used to indicate a second image quality score of the business image;

[0199] The image quality detection unit 14 is specifically used for:

[0200] Obtaining a first scoring weight for a first image quality score and a second scoring weight for a second image quality score;

[0201] Based on the first scoring weight and the second scoring weight, the first image quality score and the second image quality score are weightedly fused to obtain image quality detection data.

[0202] In one embodiment, the business feature data is obtained by performing feature extraction processing on the business image through a feature extraction processor, and the feature extraction processor includes a semantic feature extractor and a quality feature extractor;

[0203] The image processing device 1 further includes: a feature extraction unit 15;

[0204] The feature extraction unit 15 is specifically used for:

[0205] Performing semantic feature extraction processing on the business image through a semantic feature extractor to obtain semantic feature data of the business image;

[0206] Performing quality feature extraction processing on the business image through a quality feature extractor to obtain quality feature data of the business image;

[0207] The semantic feature data and the quality feature data are subjected to feature fusion processing to obtain the business feature data of the business image.

[0208] In one embodiment, the image processing device 1 further includes: a semantic feature extraction training unit 16;

[0209] The semantic feature extraction training unit 16 is specifically used for:

[0210] Acquire a first sample image for training an initial semantic feature extractor;

[0211] Performing semantic feature extraction processing on the first sample image by an initial semantic feature extractor to obtain sample semantic feature data of the first sample image;

[0212] Performing image scaling processing on the first sample image to obtain a scaled sample image corresponding to the first sample image;

[0213] Performing semantic feature extraction processing on the scaled sample image by an initial semantic feature extractor to obtain scaled semantic feature data of the scaled sample image;

[0214] Based on the sample semantic feature data and the scaled semantic feature data, parameters of the initial semantic feature extractor are adjusted, and the initial semantic feature extractor after the parameter adjustment is determined as the semantic feature extractor.

[0215] In one embodiment, the image processing device 1 further includes: a quality feature extraction training unit 17;

[0216] The quality feature extraction training unit 17 is specifically used for:

[0217] Acquire a second sample image for training an initial quality feature extractor;

[0218] Performing quality feature extraction processing on the second sample image by the initial quality feature extractor to obtain sample quality feature data of the second sample image;

[0219] Performing image segmentation on the second sample image to obtain a plurality of image slices corresponding to the second sample image, and randomly combining the plurality of image slices to obtain a combined sample image corresponding to the second sample image; an arrangement and display order of the plurality of image slices in the combined sample image is different from an arrangement and display order of the plurality of image slices in the second sample image;

[0220] Performing quality feature extraction processing on the combined sample image by an initial quality feature extractor to obtain combined quality feature data of the combined sample image;

[0221] Based on the sample quality feature data and the combined quality feature data, parameters of the initial quality feature extractor are adjusted, and the initial quality feature extractor after the parameter adjustment is determined as the quality feature extractor.

[0222] In one embodiment, each business scenario in the at least one business scenario has a corresponding scenario identifier;

[0223] The image acquisition unit 11 is specifically used for:

[0224] Acquire a business video to be subjected to image quality detection, and select a target video frame from a plurality of video frames included in the business video; the business video is associated with a target scene identifier;

[0225] Searching for a business scenario matching the target scenario identifier from at least one business scenario;

[0226] If a target business scene matching the target scene identifier is found, the target video frame is determined as a business image to be subjected to image quality detection.

[0227] In one embodiment, the image processing apparatus further includes: a business model training unit 18;

[0228] The business model training unit 18 is specifically used for:

[0229] Acquire sample training data for an initial business model; the sample training data includes a third sample image, an image quality label associated with the third sample image, and a sample standard image associated with the third sample image; the sample standard image is bound to the sample image quality data;

[0230] When the sample feature data corresponding to the third sample image and the sample standard feature data corresponding to the sample standard image are obtained, the sample feature data, the sample standard feature data and the sample image quality data are spliced ​​to obtain sample multimodal feature data for inputting into the initial business model;

[0231] Perform image quality detection on sample multimodal feature data through the initial business model to obtain a predicted image quality detection result;

[0232] Based on the predicted image quality detection results and the image quality labels, the model parameters of the initial business model are fine-tuned to obtain the target business model.

[0233] See also Fig.12 , Fig.12 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. For example, the computer device may be the above Figure 1 Server 100a in. Fig.12As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig.12 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0234] In such Fig.12 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to execute the description of the image processing method in any of the corresponding embodiments above, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.

[0235] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the image processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above embodiment of the Figure 3 and Figure 6 The description of the corresponding image processing method in the embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0236] The computer-readable storage medium may be an internal storage unit of the image processing apparatus or the computer device provided in any of the aforementioned embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0237] In addition, it should be noted here that: the embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided by any of the corresponding embodiments in the foregoing text. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0238] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0239] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0240] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0241] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a business image to be subjected to image quality detection; the business image is an image in a target business scenario; the target business scenario refers to a scene in at least one business scenario; one business scenario corresponds to one standard image set; Retrieving a target standard image matching the business image from a standard image set corresponding to the target business scenario, and acquiring target image quality data bound to the target standard image; When the service feature data of the service image and the target standard feature data of the target standard image are acquired, the service feature data, the target standard feature data and the target image quality data are spliced ​​to obtain multimodal feature data for inputting into a target service model; When the model text prompt information for the target business model is obtained, the multimodal feature data and the model text prompt information are input into the target business model, and the target business model performs image quality detection on the multimodal feature data according to the model text prompt information to obtain image quality detection data.

2. The method according to claim 1, characterized in that The retrieving a target standard image matching the business image from the standard image set corresponding to the target business scenario comprises: Performing feature extraction processing on the business image by a feature extractor to obtain business feature data of the business image; Acquire standard feature data corresponding to each standard image in the standard image set corresponding to the target business scenario; the standard feature data corresponding to any standard image is obtained by performing feature extraction processing on any standard image by the feature extractor; Determining image similarities between each standard image and the business image based on feature similarities between the business feature data and the standard feature data corresponding to each standard image; From the standard image set corresponding to the target business scene, a standard image whose image similarity with the business image satisfies a similarity condition is searched, and the searched standard image is determined as a target standard image matching the business image.

3. The method according to claim 1, characterized in that The target standard image includes M standard images, where M is a positive integer; the target standard feature data includes standard feature data corresponding to the M standard images respectively; the target image quality data includes image definition data corresponding to the M standard images respectively; The splicing of the service feature data, the target standard feature data and the target image quality data to obtain multimodal feature data for inputting a target service model includes: Performing a first splicing process on the standard feature data corresponding to each standard image in the M standard images to obtain first splicing feature data; Performing a second splicing process on the image definition data corresponding to each of the M standard images to obtain second splicing feature data; the business modality indicated by the second splicing feature data is different from the business modality indicated by the first splicing feature data; A splicing function for performing multimodal splicing processing is obtained, and the business feature data, the first splicing feature data and the second splicing feature data are spliced ​​through the splicing function to obtain multimodal feature data for inputting into a target business model.

4. The method according to claim 1, characterized in that: The step of inputting the multimodal feature data and the model text prompt information into the target business model, and performing image quality detection on the multimodal feature data according to the model text prompt information by the target business model to obtain image quality detection data includes: Constructing multimodal model prompt information for the target business model based on the multimodal feature data and the model text prompt information, and inputting the multimodal model prompt information into the target business model; The target business model performs image quality detection on the multimodal feature data according to the multimodal model prompt information to obtain a first quality detection result of the business image; The first quality detection result is determined as image quality detection data.

5. The method according to claim 1, characterized in that The step of inputting the multimodal feature data and the model text prompt information into the target business model, and performing image quality detection on the multimodal feature data according to the model text prompt information by the target business model to obtain image quality detection data includes: Acquire an image quality rule under the target business scenario; the image quality rule is configured for the target business scenario; constructing multimodal model prompt information for the target business model based on the image quality rule, the multimodal feature data and the model text prompt information, and inputting the multimodal model prompt information into the target business model; The target business model performs image quality detection on the multimodal feature data according to the multimodal model prompt information to obtain a first quality detection result of the business image; Obtaining a second quality detection result for the business image according to the image quality rule and the first quality detection result; Image quality detection data is determined based on the first quality detection result and the second quality detection result.

6. The method according to claim 5, characterized in that The image quality rule includes N sub-rules, where N is a positive integer; The obtaining, by using the image quality rule and the first quality detection result, a second quality detection result for the business image includes: Determine a target sub-rule matching the business image from the N sub-rules; The target sub-rule is called to adjust the quality result of the first quality detection result to obtain a second quality detection result for the business image.

7. The method according to claim 5, characterized in that The first quality detection result is used to indicate a first image quality score of the business image, and the second quality detection result is used to indicate a second image quality score of the business image; The determining of image quality detection data based on the first quality detection result and the second quality detection result includes: Acquire a first scoring weight for the first image quality score and a second scoring weight for the second image quality score; Based on the first scoring weight and the second scoring weight, the first image quality score and the second image quality score are weightedly fused to obtain image quality detection data.

8. The method according to claim 1, characterized in that The business feature data is obtained by performing feature extraction processing on the business image through a feature extraction processor, wherein the feature extraction processor includes a semantic feature extractor and a quality feature extractor; The method further comprises: Performing semantic feature extraction processing on the business image by the semantic feature extractor to obtain semantic feature data of the business image; Performing quality feature extraction processing on the business image by the quality feature extractor to obtain quality feature data of the business image; Feature fusion processing is performed on the semantic feature data and the quality feature data to obtain business feature data of the business image.

9. The method according to claim 8, characterized in that The method further comprises: Acquire a first sample image for training an initial semantic feature extractor; Performing semantic feature extraction processing on the first sample image by the initial semantic feature extractor to obtain sample semantic feature data of the first sample image; Performing image scaling processing on the first sample image to obtain a scaled sample image corresponding to the first sample image; Performing semantic feature extraction processing on the scaled sample image by the initial semantic feature extractor to obtain scaled semantic feature data of the scaled sample image; Based on the sample semantic feature data and the scaled semantic feature data, parameters of the initial semantic feature extractor are adjusted, and the initial semantic feature extractor after the parameter adjustment is determined as the semantic feature extractor.

10. The method according to claim 8, characterized in that The method further comprises: Acquire a second sample image for training an initial quality feature extractor; Performing quality feature extraction processing on the second sample image by the initial quality feature extractor to obtain sample quality feature data of the second sample image; Performing image segmentation on the second sample image to obtain a plurality of image slices corresponding to the second sample image, and randomly combining the plurality of image slices to obtain a combined sample image corresponding to the second sample image; an arrangement and display order of the plurality of image slices in the combined sample image is different from an arrangement and display order of the plurality of image slices in the second sample image; Performing quality feature extraction processing on the combined sample image by the initial quality feature extractor to obtain combined quality feature data of the combined sample image; Based on the sample quality feature data and the combined quality feature data, parameters of the initial quality feature extractor are adjusted, and the initial quality feature extractor after the parameter adjustment is determined as the quality feature extractor.

11. The method according to claim 1, characterized in that: Each business scenario in the at least one business scenario has a corresponding scenario identifier; The obtaining of the service image to be subjected to image quality detection comprises: Acquire a business video to be subjected to image quality detection, and select a target video frame from a plurality of video frames included in the business video; the business video is associated with a target scene identifier; Searching for a business scenario matching the target scenario identifier from the at least one business scenario; If a target business scene matching the target scene identifier is found, the target video frame is determined as a business image to be subjected to image quality detection.

12. The method according to claim 1, characterized in that The method further comprises: Acquire sample training data for an initial business model; the sample training data includes a third sample image, an image quality label associated with the third sample image, and a sample standard image associated with the third sample image; the sample standard image is bound with sample image quality data; When the sample feature data corresponding to the third sample image and the sample standard feature data corresponding to the sample standard image are acquired, the sample feature data, the sample standard feature data and the sample image quality data are spliced ​​to obtain sample multimodal feature data for inputting the initial business model; Performing image quality detection on the sample multimodal feature data through an initial business model to obtain a predicted image quality detection result; Based on the predicted image quality detection result and the image quality label, model parameters of the initial business model are fine-tuned to obtain a target business model.

13. An image processing device, characterized in that: The device comprises: An image acquisition unit, used to acquire a business image to be subjected to image quality detection; the business image is an image in a target business scenario; the target business scenario refers to a scene in at least one business scenario; one business scenario corresponds to one standard image set; An image retrieval unit is used to retrieve a target standard image matching the business image from a standard image set corresponding to the target business scenario, and obtain target image quality data bound to the target standard image; a feature data splicing unit, configured to, when acquiring the service feature data of the service image and the target standard feature data of the target standard image, splice the service feature data, the target standard feature data and the target image quality data to obtain multimodal feature data for inputting into a target service model; An image quality detection unit is used to input the multimodal feature data and the model text prompt information into the target business model when the model text prompt information for the target business model is obtained, and the target business model performs image quality detection on the multimodal feature data according to the model text prompt information to obtain image quality detection data.

14. A computer device, characterized in that: including memory and processor; The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the method according to any one of claims 1 to 12 when executed by a processor.

Images