An image recognition method and device, electronic equipment and storage medium

Abstract

The specification discloses a method and device for image recognition, electronic equipment and storage medium. A pre-trained multi-modal pre-training model and a classification model are deployed on a server, and the number of model parameters contained in the multi-modal pre-training model is greater than the number of model parameters contained in the classification model. First, an image to be recognized is obtained. Second, the image to be recognized is input into the classification model, and whether the image to be recognized is a violation image is determined based on the classification result output by the classification model. Finally, if it is determined that the image to be recognized is a violation image, the image to be recognized is further input into the multi-modal pre-training model for violation recognition. The method can reduce the operation resources required for image recognition and improve the efficiency of image recognition.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for image recognition. Background Technology

[0002] With the rapid development of the internet, images have become one of the main forms of information dissemination for internet platform users due to their intuitiveness and ability to carry a large amount of information. To attract traffic, malicious actors generate or spread a large number of illegal images, such as pornographic and violent images. These illegal images can cause serious social impacts or negatively affect the normal operation of internet platforms. Therefore, internet platforms need to identify user-uploaded images to prevent users from spreading illegal images.

[0003] Currently, a common approach is to train a multimodal pre-trained model on a large-scale dataset, and then use this model to recognize user-uploaded images. However, multimodal pre-trained models require significant computational resources, resulting in low efficiency in image recognition.

[0004] Therefore, how to reduce the computational resources required for image recognition and improve its efficiency is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a method, apparatus, storage medium, and electronic device for image recognition, in order to reduce the computing resources required for image recognition and improve the efficiency of image recognition.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides an image recognition method applied to a server. The server deploys a pre-trained multimodal pre-trained model and a classification model. The multimodal pre-trained model contains a greater number of model parameters than the classification model, including:

[0008] Acquire the image to be recognized;

[0009] The image to be identified is input into the classification model, and the classification result output by the classification model is used to determine whether the image to be identified is a violation image;

[0010] If the image to be identified is determined to be a violation image, the image to be identified is further input into the multimodal pre-trained model for violation identification.

[0011] Optionally, the size of the input image corresponding to the classification model is smaller than the size of the input image of the multimodal pre-trained model;

[0012] The process involves inputting the image to be identified into the classification model, and determining whether the image to be identified is a violation image based on the classification result output by the classification model, including:

[0013] The image to be identified is scaled according to the input image size corresponding to the classification model. The scaled image to be identified is then input into the classification model, and the classification result output by the classification model is used to determine whether the image to be identified is a violation image.

[0014] Optionally, the server deploys multiple classification models with different input image sizes.

[0015] According to the input image size corresponding to the classification model, the image to be identified is scaled, the scaled image to be identified is input into the classification model, and the classification result output by the classification model is used to determine whether the image to be identified is a violation image, including:

[0016] The multiple classification models are sorted in ascending order of the input image size;

[0017] According to the input image size corresponding to the first classification model among the sorted multiple classification models, the image to be identified is scaled, the scaled image to be identified is input into the first classification model, and the classification result output by the first classification model is used to determine whether the image to be identified is a violation image.

[0018] If so, continue scaling the image to be identified according to the input image size corresponding to the second classification model among the sorted multiple classification models. Input the scaled image to be identified into the second classification model and determine whether the image to be identified is a violation image based on the classification result output by the second classification model. Continue in this manner until the image to be identified is scaled according to the input image size corresponding to the last classification model among the sorted multiple classification models, and the scaled image to be identified is input into the last classification model.

[0019] Optionally, if the image to be identified is determined to be a violation image, the image to be identified is further input into the multimodal pre-trained model for violation identification, including:

[0020] Based on the classification result output by the last classification model, determine whether the image to be identified is a violation image; if so, further input the image to be identified into the multimodal pre-trained model for violation identification.

[0021] Optionally, the method further includes:

[0022] If any of the multiple classification models is determined to output a classification result that the image to be identified is a non-violation image, the scaled image to be identified will no longer be input into other classification models located after the sorting position of that classification model.

[0023] Optionally, the multimodal pre-trained model includes: a visual question answering model;

[0024] Inputting the image to be identified into the multimodal pre-trained model for violation identification includes:

[0025] Obtain multiple text questions for violation detection;

[0026] The image to be identified and the multiple text questions are input into a visual question answering model. Based on the response texts to the multiple text questions, it is determined whether the image to be identified is a violation image.

[0027] Optionally, the classification model is trained and optimized with the goal of minimizing the false negative rate, whereby the false negative rate is the ratio of the number of non-violation images to the total number of non-violation images.

[0028] Optionally, before inputting the image to be identified into the classification model, the method further includes:

[0029] Determine whether the user blacklist contains user information for the image to be identified; if so, determine that the image to be identified is a violation image.

[0030] Determine whether the user whitelist contains user information for the image to be identified. If so, determine that the image to be identified is a non-violation image.

[0031] Optionally, before inputting the image to be identified into the classification model, the method further includes:

[0032] Edge detection is performed on the image to be identified to determine the edge information of the image targets contained in the image to be identified;

[0033] The determination of whether an image to be identified is a violation image is based on the amount of edge information of the image target contained in the image to be identified.

[0034] Optionally, the visual question answering model includes: the BLIP model.

[0035] Optionally, the image to be identified includes: an image drawn by a user using an AI-based drawing program.

[0036] This specification provides an image recognition apparatus applied to a server. The server deploys a pre-trained multimodal pre-trained model and a classification model. The multimodal pre-trained model contains a greater number of model parameters than the classification model, including:

[0037] The acquisition module is used to acquire the image to be recognized;

[0038] An execution module is used to input the image to be identified into the classification model and determine whether the image to be identified is a violation image based on the classification result output by the classification model.

[0039] The input module is used to further input the image to be identified into the multimodal pre-trained model for violation identification if it is determined that the image to be identified is a violation image.

[0040] This specification provides an electronic device, including a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus;

[0041] The memory stores machine-readable instructions, and the processor executes the image recognition method described above by invoking the machine-readable instructions.

[0042] This specification provides a machine-readable storage medium storing machine-readable instructions that, when called and executed by a processor, implement the above-described image recognition method.

[0043] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0044] In the image recognition method provided in this specification, the multimodal pre-trained model contains a greater number of model parameters than the classification model. This method can input the image to be recognized into the classification model and determine whether the image to be recognized is a violation image based on the classification result output by the classification model, thereby identifying images that are easier to recognize and filtering out images that are more difficult to recognize. If the image to be recognized is determined to be a violation image, the more difficult images to recognize are further input into the multimodal pre-trained model for violation recognition, thereby reducing the computational resources required for image recognition and improving the efficiency of image recognition. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart illustrating an image recognition method as an exemplary embodiment;

[0047] Figure 2 This is an exemplary embodiment illustrating a flowchart of multiple classification models recognizing an image to be identified;

[0048] Figure 3 This is an exemplary embodiment illustrating a flowchart for determining whether an image to be identified is a violation image;

[0049] Figure 4 This is a schematic diagram of the structure of an electronic device containing an image recognition apparatus, as shown in an exemplary embodiment.

[0050] Figure 5 This is a block diagram illustrating an image recognition device as an exemplary embodiment. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0052] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0053] To enable those skilled in the art to better understand the technical solutions in the embodiments of this specification, the relevant technologies involved in the embodiments of this specification will be briefly described below.

[0054] A multimodal pre-trained model refers to a deep learning model that is pre-trained on a large-scale dataset using self-supervised learning methods. This allows it to effectively learn representations of multiple modalities (such as text, images, and speech) and achieve specific applications through fine-tuning or task-specific training. Multimodal pre-trained models have achieved good performance on many classic multimodal tasks, such as visual question answering, image caption generation, and image-text retrieval.

[0055] In practical applications, multimodal pre-trained models are typically used to recognize user-uploaded images. However, multimodal pre-trained models require significant computational resources, resulting in relatively low image recognition efficiency.

[0056] Based on this, this specification proposes a technical solution that first inputs the model to be recognized into a classification model with a small number of model parameters to identify the image to be recognized that is easier to recognize, and then inputs the image to be recognized that is more difficult to recognize into a multimodal pre-trained model for recognition. This reduces the computational resources required for image recognition and improves the efficiency of image recognition.

[0057] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart illustrating an exemplary embodiment of an image recognition method, specifically including the following steps:

[0059] S100: Acquire the image to be recognized.

[0060] In the embodiments of this specification, the image recognition method is applied to the server. The physical carrier of the server can be a server, a server cluster, etc. For ease of description, the image recognition method provided in this specification will be described below with the server as the execution subject only.

[0061] The server-side deployment includes a pre-trained multimodal pre-trained model and a classification model. The multimodal pre-trained model contains more model parameters than the classification model.

[0062] In the embodiments described in this specification, the server can obtain the image to be identified. The image to be identified mentioned here includes: an image drawn by the user using an AI-based drawing program.

[0063] S102: Input the image to be identified into the classification model, and determine whether the image to be identified is a violation image based on the classification result output by the classification model.

[0064] S104: If the image to be identified is determined to be a violation image, the image to be identified is further input into the multimodal pre-trained model for violation identification.

[0065] In the embodiments described in this specification, the server can train the classification model before inputting the image to be recognized into the classification model. There are various methods for the server to train the classification model. For example, the server can prepare a dataset according to service requirements and divide the images in the dataset into a training set and a validation set according to a set ratio. Secondly, the server can scale the image to be recognized according to the input image size corresponding to the classification model, and input the scaled image into the classification model. The goal is to minimize the deviation between the classification result output by the classification model and the actual result, thereby training the classification model. This specification does not limit the method of training the classification model, nor does it limit the model results of the classification model. The service requirements mentioned here can refer to requirements such as recognizing pornographic images or recognizing images of public figures.

[0066] In practical applications, multimodal pre-trained models contain a large number of model parameters and achieve high image recognition accuracy. However, they require more computational resources and have lower image recognition efficiency. Conversely, classification models contain fewer model parameters and achieve lower image recognition accuracy, but they require fewer computational resources and have higher image recognition efficiency.

[0067] Therefore, the server can use a classification model to identify easily identifiable non-violation images in the image to be identified, thereby reducing the computational resources required for image recognition and improving the efficiency of image recognition. Then, the non-violation images that are not easy to identify in the image to be identified are input into a multimodal pre-trained model for violation identification, so as to correctly identify the image to be identified.

[0068] In the embodiments described in this specification, the server can input the image to be identified into the classification model and determine whether the image to be identified is a violation image based on the classification result output by the classification model.

[0069] If the image to be identified is determined to be a violation image, it is further input into a multimodal pre-trained model for violation identification.

[0070] To effectively reduce computational complexity and improve image recognition efficiency, the classification model is trained on a training set consisting of images with relatively small image sizes. Correspondingly, the input image size for the classification model is smaller than the input image size for the multimodal pre-trained model. Therefore, the server needs to scale the image to be recognized before inputting it into the classification model.

[0071] In the embodiments of this specification, the server can scale the image to be identified according to the input image size corresponding to the classification model, input the scaled image to be identified into the classification model, and determine whether the image to be identified is a violation image based on the classification result output by the classification model.

[0072] Furthermore, the classification model can be trained and optimized with the goal of minimizing the false negative rate. The false negative rate refers to the ratio of the number of illegal images classified as non-illegal images to the total number of illegal images. For example, if the number of illegal images is determined to be 100, and the classification model only identifies 90 illegal images, missing 10, then the false negative rate is 10%.

[0073] This shows that, while ensuring the false negative rate of the classification model, if the image to be identified is determined to be a non-violation image, even if the classification model incorrectly identifies the image to be identified as a violation image, a final violation identification will be performed through the multimodal pre-trained model, thus correctly identifying the image to be identified as a non-violation image.

[0074] In practical applications, a single classification model can only identify a limited number of non-violation images. This means that multimodal pre-trained models still need to process a large number of images for violation identification, resulting in minimal improvement in image recognition efficiency. Therefore, multiple classification models can be deployed on the server side to increase the number of non-violation images identified.

[0075] However, processing multiple classification models simultaneously consumes significant computational resources. Therefore, the server can sort the input images according to their size and input them sequentially. If an image is determined to be a violation, it is moved to the next classification model for processing. If an image is determined to be non-violation, subsequent models stop processing it and output "non-violation."

[0076] In the embodiments described in this specification, multiple classification models with different input image sizes are deployed on the server. The server can sort the multiple classification models in ascending order of input image size. The image size mentioned here can refer to the number of pixels in the horizontal and vertical directions of the image, usually expressed as width and height. For example, a 16*16 image, a 64*64 image, etc.

[0077] Then, the server can scale the image to be identified according to the input image size corresponding to the first classification model among the sorted classification models. The scaled image is then input into the first classification model, and the server determines whether the image is a violation based on the classification result output by the first model. The server can scale the image to be identified according to a preset scaling ratio, or it can scale it according to the input image size corresponding to the classification model. Furthermore, the server can represent the number of classification models by the number of scaling categories.

[0078] If so, continue scaling the image to be identified according to the input image size of the second classification model among the sorted classification models. Input the scaled image to be identified into the second classification model and determine whether the image to be identified is a violation based on the classification result output by the second classification model. Continue in this manner until the image to be identified is scaled according to the input image size of the last classification model among the sorted classification models, and the scaled image to be identified is input into the last classification model.

[0079] Next, the server can determine whether the image to be identified is a violation image based on the classification result output by the last classification model. If so, the image to be identified is further input into the multimodal pre-trained model for violation identification.

[0080] If any one of the multiple classification models is determined to output a classification result indicating that the image to be identified is a non-violation image, the scaled image to be identified will not be input into any other classification model located after that model in the ranking. Specifically, as follows... Figure 2 As shown.

[0081] Figure 2 This is an exemplary embodiment illustrating a flowchart of multiple classification models recognizing an image to be recognized.

[0082] exist Figure 2 In this process, the scaled image to be identified corresponding to the first classification model is input into the first classification model to determine whether the image to be identified is a violation image. If not, the image to be identified is output as a non-violation image. If yes, the scaled image to be identified corresponding to the second classification model is input into the second classification model to determine whether the image to be identified is a violation image. And so on, the scaled image to be identified corresponding to the Nth classification model is input into the Nth classification model to determine whether the image to be identified is a violation image.

[0083] This demonstrates that, while maintaining the false negative rate of the classification model, even if one of the multiple classification models incorrectly identifies a non-violation image as a violation image, subsequent classification models can still identify the non-violation image again. Furthermore, a final violation identification is performed using a multimodal pre-trained model. This approach reduces the computational resources required for image recognition and improves the efficiency of image recognition while ensuring the accuracy of the image being identified.

[0084] In the embodiments described in this specification, the multimodal pre-trained model includes a visual question-answering model. The server can obtain multiple text questions for violation detection. The text questions mentioned here can be text pre-entered by the user.

[0085] Different service requirements correspond to different text questions. For example, if the service requirement is to detect pornographic images, text questions might include: Does the image contain a certain male genitalia? Does the image contain a certain female genitalia? As another example, if the service requirement is to detect images of public figures, text questions might include: Does the image contain a flag representing a certain country or region? Does the image contain a certain public figure? This specification does not limit the specific description of the text questions.

[0086] Then, the server can input the image to be identified and multiple text questions into the visual question-answering model. Based on the response texts to the multiple text questions, it determines whether the image to be identified is a violation image. The response texts mentioned here can be affirmative or negative answers, such as yes or no, exist or do not exist, correct or incorrect, etc. This specification does not limit the specific description of the response texts.

[0087] Specifically, if at least one positive answer is found in the responses to multiple text questions, the image to be identified is determined to be a violation image. If all responses to multiple text questions are negative answers, the image to be identified is determined to be a non-violation image.

[0088] Furthermore, the server can construct a list of multiple text questions. The image to be recognized and the list of text questions are input into the visual question-answering model. Following the order of the text questions in the list, the server sequentially determines the response text for each text question. If the response text for a text question is determined to be a negative answer, the server proceeds to determine the response text for the next text question. If the response text for any text question in the list is determined to be a positive answer, the image to be recognized is identified as a violation image, and no further response texts are determined for subsequent text questions.

[0089] In practical applications, some users frequently send inappropriate images or engage in other violations. The server can add these violating users to a blacklist, directly identifying the images to be recognized from these users as inappropriate. Conversely, some users are trusted, and the server can add these trusted users to a whitelist, directly identifying the images to be recognized from these users as compliant. This improves the efficiency of image recognition.

[0090] In the embodiments described in this specification, the server may pre-store a user blacklist and a user whitelist. The server can determine whether the user blacklist contains user information for the image to be identified; if so, the image to be identified is determined to be a violation image.

[0091] Of course, the server can also determine whether the user whitelist contains the user information of the image to be identified. If so, it determines that the image to be identified is a non-violation image.

[0092] In practical applications, some user-uploaded images contain too little information, such as pure white or pure black images. Recognizing these images would waste computational resources. Therefore, the server can first perform edge detection on the image to be recognized to determine the edge information and thus whether the image is a violation.

[0093] In the embodiments of this specification, the server can perform edge detection on the image to be identified to determine the edge information of the image target contained in the image to be identified.

[0094] Then, the server can determine whether the image to be identified is a violation image based on the amount of edge information of the image targets contained in the image to be identified.

[0095] Specifically, the logic used by the server to determine whether an image to be identified is a violation varies depending on the service requirements. For example, when the number of edge information points of the image target in the image to be identified is small, it is considered a meaningless image uploaded by the user, and the server can directly determine that the image to be identified is a violation. Under this service requirement, if the number of edge information points of the image target in the image to be identified is less than a set number, the image to be identified is determined to be a violation.

[0096] For example, when the image to be identified contains a small amount of edge information of the image targets, no non-compliant images will appear in the image, and the server can directly determine that the image to be identified is a non-compliant image. Under this service requirement, if the amount of edge information of the image targets in the image to be identified is less than a set amount, the image to be identified is determined to be a non-compliant image.

[0097] There are various edge detection algorithms, such as the Canny algorithm and the Roberts operator. This specification does not limit the edge detection algorithm.

[0098] It should be noted that visual question answering models include the BLIP model.

[0099] In the embodiments described in this specification, the server can determine whether the image to be identified is a violation image through multiple steps. Specifically, as follows... Figure 3 As shown.

[0100] Figure 3 This is an exemplary embodiment illustrating a flowchart for determining whether an image to be identified is a violation image.

[0101] exist Figure 3 In the process, the server can determine whether the user information of the image to be identified is included in the user blacklist and user whitelist. If not, edge detection is performed on the image to be identified to determine whether the image to be identified is a violation image.

[0102] If yes, input the scaled image to be identified corresponding to the first classification model into the first classification model to determine whether the image to be identified is a violation image. If no, output the image to be identified as a non-violation image. If yes, input the scaled image to be identified corresponding to the second classification model into the second classification model to determine whether the image to be identified is a violation image. And so on, input the scaled image to be identified corresponding to the Nth classification model into the Nth classification model to determine whether the image to be identified is a violation image.

[0103] If so, the image to be identified is further input into the multimodal pre-trained model for violation identification to determine whether the image to be identified is a violation image.

[0104] As can be seen from the above method, this method can input the image to be identified into a classification model, and determine whether the image to be identified is a violation image based on the classification result output by the model. This allows for the identification of images with lower recognition difficulty and the filtering of images with higher recognition difficulty. If the image to be identified is determined to be a violation image, the images with higher recognition difficulty are further input into a multimodal pre-trained model for violation identification, thereby reducing the computational resources required for image recognition and improving the efficiency of image recognition.

[0105] Corresponding to the embodiments of the image recognition method described above, this specification also provides an embodiment of an image recognition apparatus.

[0106] Please see Figure 4 , Figure 4 This is a structural diagram of an electronic device containing an image recognition apparatus, as illustrated in an exemplary embodiment. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other necessary hardware. One or more embodiments of this specification can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into memory 408 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0107] Please see Figure 5 , Figure 5 This is a block diagram illustrating an image recognition apparatus as an exemplary embodiment. This image recognition apparatus can be applied to, for example... Figure 4 The electronic device shown implements the technical solution of this specification. The image recognition device may include:

[0108] The acquisition module 500 is used to acquire the image to be recognized;

[0109] The input module 502 is used to input the image to be identified into the classification model, and determine whether the image to be identified is a violation image based on the classification result output by the classification model;

[0110] The identification module 504 is used to further input the image to be identified into the multimodal pre-trained model for violation identification if it is determined that the image to be identified is a violation image.

[0111] Optionally, the input image size corresponding to the classification model is smaller than the input image size of the multimodal pre-trained model. The input module 502 is specifically used to scale the image to be identified according to the input image size corresponding to the classification model, input the scaled image to be identified into the classification model, and determine whether the image to be identified is a violation image based on the classification result output by the classification model.

[0112] Optionally, the server deploys multiple classification models with different input image sizes. The input module 502 is specifically used to sort the multiple classification models in ascending order of the input image size, scale the image to be identified according to the input image size of the first classification model in the sorted multiple classification models, input the scaled image to be identified into the first classification model, and determine whether the image to be identified is a violation image based on the classification result output by the first classification model. If it is, the system continues to scale the image to be identified according to the input image size of the second classification model in the sorted multiple classification models, input the scaled image to be identified into the second classification model, and determine whether the image to be identified is a violation image based on the classification result output by the second classification model, and so on, until the image to be identified is scaled according to the input image size of the last classification model in the sorted multiple classification models, and the scaled image to be identified is input into the last classification model.

[0113] Optionally, the recognition module 504 is specifically used to determine whether the image to be recognized is a violation image based on the classification result output by the last classification model; if so, the image to be recognized is further input into the multimodal pre-trained model for violation recognition.

[0114] Optionally, the recognition module 504 is further configured to, if it is determined that any one of the multiple classification models outputs a classification result for the image to be recognized as a non-violation image, not to input the scaled image to be recognized into other classification models located after the sorting position of that classification model.

[0115] Optionally, the multimodal pre-trained model includes a visual question answering model. The recognition module 504 is specifically used to acquire multiple text questions for violation recognition, input the image to be recognized and the multiple text questions into the visual question answering model, and determine whether the image to be recognized is a violation image based on the response text to the multiple text questions.

[0116] Optionally, the classification model is trained and optimized with the goal of minimizing the false negative rate, whereby the false negative rate is the ratio of the number of non-violation images to the total number of non-violation images.

[0117] Optionally, the acquisition module 500 is further configured to determine whether the user blacklist contains user information of the image to be identified; if so, determine that the image to be identified is a violation image; and determine whether the user whitelist contains user information of the image to be identified; if so, determine that the image to be identified is a non-violation image.

[0118] Optionally, the acquisition module 500 is further configured to perform edge detection on the image to be identified, determine the edge information of the image targets contained in the image to be identified, and determine whether the image to be identified is a violation image based on the number of edge information of the image targets contained in the image to be identified.

[0119] Optionally, the visual question answering model includes: the BLIP model.

[0120] Optionally, the image to be identified includes: an image drawn by a user using an AI-based drawing program.

[0121] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0122] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0123] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0124] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0126] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0127] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0128] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0129] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0130] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0131] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0132] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. An image recognition method, said method being applied to a server, wherein a pre-trained multimodal pre-trained model and multiple classification models are deployed on the server, the multiple classification models comprising: Multiple classification models corresponding to input image sizes sorted from smallest to largest; The multimodal pre-trained model contains a greater number of model parameters than the classification model, and the input image size corresponding to the classification model is smaller than the input image size of the multimodal pre-trained model. The method includes: Acquire the image to be recognized; The process of inputting the image to be identified into the classification model and determining whether the image to be identified is a violation image based on the classification result output by the classification model includes: scaling the image to be identified according to the input image size corresponding to the first classification model among the multiple classification models; inputting the scaled image to be identified into the first classification model; and determining whether the image to be identified is a violation image based on the classification result output by the first classification model; if so, scaling the image to be identified according to the input image size corresponding to the second classification model among the multiple classification models; inputting the scaled image to be identified into the second classification model; and determining whether the image to be identified is a violation image based on the classification result output by the second classification model, and so on, until scaling the image to be identified according to the input image size corresponding to the last classification model among the multiple classification models; and inputting the scaled image to be identified into the last classification model; if any classification model among the multiple classification models outputs a classification result indicating that the image to be identified is a non-violation image, the scaled image to be identified is no longer input into other classification models located after the sorting position of that classification model. If the image to be identified is determined to be a violation image, the image to be identified is further input into the multimodal pre-trained model for violation identification; wherein, the multimodal pre-trained model includes a visual question answering model, and the input of the visual question answering model includes a text question for violation identification and the image to be identified.

2. The method as described in claim 1, wherein if the image to be identified is determined to be a violation image, the image to be identified is further input into the multimodal pre-trained model for violation identification, comprising: Based on the classification result output by the last classification model, determine whether the image to be identified is a violation image; If so, the image to be identified is further input into the multimodal pre-trained model for violation identification.

3. The method as described in claim 1, wherein the image to be identified is input into the multimodal pre-trained model for violation identification, comprising: Obtain multiple text questions for violation detection; The image to be identified and the multiple text questions are input into a visual question answering model. Based on the response texts to the multiple text questions, it is determined whether the image to be identified is a violation image.

4. The method as described in claim 1, wherein the classification model is optimized and trained with the goal of minimizing the false negative rate, wherein the false negative rate refers to the ratio of the number of non-violation images to the total number of non-violation images.

5. The method of claim 1, wherein before inputting the image to be identified into the classification model, the method further comprises: Determine whether the user blacklist contains user information for the image to be identified; if so, determine that the image to be identified is a violation image. Determine whether the user whitelist contains the user information of the image to be identified. If so, determine that the image to be identified is a non-violation image.

6. The method of claim 1, wherein before inputting the image to be identified into the classification model, the method further comprises: Edge detection is performed on the image to be identified to determine the edge information of the image targets contained in the image to be identified; The determination of whether an image to be identified is a violation image is based on the amount of edge information of the image target contained in the image to be identified.

7. The method of claim 1, wherein the visual question answering model comprises: BLIP model.

8. The method of claim 1, wherein the image to be identified comprises: Images drawn by users using AI-based drawing programs.

9. An image recognition apparatus, the apparatus being applied to a server, the server having deployed a pre-trained multimodal pre-trained model and multiple classification models, the multiple classification models comprising: Multiple classification models corresponding to input image sizes sorted from smallest to largest; The multimodal pre-trained model contains a greater number of model parameters than the classification model, and the input image size corresponding to the classification model is smaller than the input image size of the multimodal pre-trained model. The device includes: The acquisition module is used to acquire the image to be recognized; An input module is used to input the image to be identified into the classification model and determine whether the image to be identified is a violation image based on the classification result output by the classification model. This includes: scaling the image to be identified according to the input image size corresponding to the first classification model among the sorted plurality of classification models; inputting the scaled image to be identified into the first classification model; and determining whether the image to be identified is a violation image based on the classification result output by the first classification model. If it is, the module continues to scale the image to be identified according to the input image size corresponding to the second classification model among the sorted plurality of classification models; inputting the scaled image to be identified into the second classification model; and determining whether the image to be identified is a violation image based on the classification result output by the second classification model. This process continues until the image to be identified is scaled according to the input image size corresponding to the last classification model among the sorted plurality of classification models, and the scaled image to be identified is input into the last classification model. The identification module is used to further input the image to be identified into the multimodal pre-trained model for violation identification if it is determined that the image to be identified is a violation image; wherein, the multimodal pre-trained model includes a visual question answering model, and the input of the visual question answering model includes a text question for violation identification and the image to be identified.

10. An electronic device, comprising a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 8 by invoking the machine-readable instructions.

11. A machine-readable storage medium storing machine-readable instructions that, when invoked and executed by a processor, implement the method of any one of claims 1 to 8.

Images