Image detection method and device, electronic equipment and storage medium

Abstract

The invention provides an image detection method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the fields of deep learning, image processing and the like. According to the specific implementation scheme, image features of a to-be-detected image are determined; determining the similarity between the image feature and each reference feature in response to the detection that both the text feature and the image feature of the at least one reference text meet the matching condition; wherein the reference features are determined according to features of a reference image including additional information; and determining a detection result of the to-be-detected image according to the similarity between the image feature and each reference feature, the detection result indicating whether the to-be-detected image includes additional information and the type of the additional information when the to-be-detected image includes the additional information.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning and image processing. More specifically, the present disclosure provides an image detection method, device, electronic device, storage medium, and computer program product. Background Art

[0002] Some users insert additional information such as QR codes and mobile phone numbers into pictures to attract traffic. These additional information not only violates the user rules of the platform, but is also easily used by black and gray industries to conduct online fraud and illegal traffic diversion. Therefore, it is necessary to review pictures to maintain platform rules and improve security. Summary of the invention

[0003] The present disclosure provides an image detection method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, there is provided an image detection method, comprising: determining image features of an image to be detected; in response to detecting that both text features and image features of at least one reference text satisfy matching conditions, determining the similarity between the image features and each reference feature; wherein the reference feature is determined based on features of a reference image including additional information; and determining a detection result of the image to be detected based on the similarity between the image features and each reference feature, wherein the detection result indicates: whether the image to be detected includes additional information, and the type of the additional information if the additional information is included.

[0005] According to another aspect of the present disclosure, there is provided an image detection device, comprising: an image feature determination module, a similarity determination module and a first result determination module. The image feature determination module is used to determine the image features of the image to be detected. The similarity determination module is used to determine the similarity between the image features and each reference feature in response to detecting that both the text features and the image features of at least one reference text satisfy a matching condition; wherein the reference feature is determined based on the features of the reference image including additional information. The first result determination module is used to determine the detection result of the image to be detected based on the similarity between the image features and the each reference feature, wherein the detection result indicates: whether the image to be detected includes additional information, and the type of the additional information if the additional information is included.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an application scenario of the image detection method and device according to an embodiment of the present disclosure;

[0012] Figure 2 is a schematic flow chart of an image detection method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic flow chart of an image detection method according to another embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of an image detection method according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of an image detection method according to another embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram of a deep learning model training method according to an embodiment of the present disclosure;

[0017] Figure 7 is a schematic structural block diagram of an image detection device according to an embodiment of the present disclosure; and

[0018] Figure 8 It is a structural block diagram of an electronic device used to implement the image detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0021] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0022] In related technologies, OCR (Optical Character Recognition) technology or QR code detection technology can be used to detect whether an image contains additional information for traffic diversion. However, the above technical solutions are not robust and generalizable enough. When the additional information for traffic diversion in the image is hidden by means of artistic characters, homophonic characters, homophones, emoticons, artificial intelligence art QR codes, handwriting, etc., detection failure is likely to occur.

[0023] The disclosed embodiment provides an image detection method, which first determines whether the image features of the image to be detected and the text features of the reference text meet the matching conditions, and then determines the detection result of the image to be detected based on the similarity between the image features and the reference features, and the above-mentioned reference features are determined based on the features of the reference image including additional information. By adopting the above-mentioned technical scheme, the detection result of the image to be detected can be determined based on multimodal data such as reference text and reference image, so as to effectively identify and classify various forms of additional information used for drainage, and effectively improve the accuracy and robustness of drainage information detection. In addition, it is also possible to automatically review the pictures uploaded by users, improve the review efficiency, and reduce the workload of manual review.

[0024] The image detection method provided by the embodiment of the present disclosure can be applied to a variety of products or projects that require content review and drainage detection of pictures, thereby improving the platform's detection ability for cheating users and ensuring the healthy ecology of the platform. For example, it can be applied to social media platforms to detect additional information in pictures uploaded by users to prevent improper promotion and drainage behavior. For another example, it can be applied to e-commerce platforms to review product pictures uploaded by sellers, prevent merchants from guiding users to other platforms for transactions through pictures, and avoid users being deceived by bad merchants. For another example, it can be applied to advertising review systems, used for advertising companies or platforms, to review the content in advertising pictures to ensure legality and compliance. For another example, it can be applied to content review systems such as news websites and forums to review the content of pictures uploaded by users to prevent malicious drainage through pictures. For another example, it can be applied to network security systems, used for various network security companies or platforms, to detect and prevent malicious information spread through pictures, and improve network security. For another example, it can be applied to education and training, and applied in educational platforms, training institutions and other occasions to prevent the spread of bad information through pictures.

[0025] The technical solution provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Figure 1 It is a schematic diagram of an application scenario of the image detection method and device according to an embodiment of the present disclosure.

[0027] It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0028] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0030] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as determining the detection results based on the images uploaded by the users, and determining whether to display the images normally or block the images to other users based on the detection results, etc.) to the terminal device.

[0031] It should be noted that the image detection method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image detection device provided in the embodiment of the present disclosure can generally be set in the server 105. The image detection method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image detection device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] Figure 2 is a schematic flow chart of an image detection method according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, the image detection method 200 may include operations S210 to S230.

[0035] In operation S210 , image features of an image to be detected are determined.

[0036] For example, a deep learning model may be pre-trained, and the image features of the image to be detected may be extracted by the deep learning model. For example, the deep learning model includes an image encoder, and the image encoder is used to extract the image features of the image to be detected. The deep learning model may be, for example, a CLIP (Contrastive Language–Image Pre-training) model.

[0037] In operation S220, in response to detecting that both text features and image features of at least one reference text satisfy a matching condition, similarities between the image features and respective reference features are determined; wherein the reference features are determined based on features of the reference image including additional information.

[0038] For example, the reference text may be text related to the purpose of traffic diversion. For example, the reference text is a character describing information such as a mobile phone number and a website used for traffic diversion. For another example, the reference text is a text describing a picture used for traffic diversion, such as "a picture containing a mobile phone number" or "a picture containing a QR code". For another example, the reference text describes the regularity or characteristics of the information used for traffic diversion, such as examples describing the number of digits and the first few digits. For example, the deep learning model may also include a text encoder, and the text features of the reference text may be extracted by the text encoder.

[0039] For example, the matching condition may include: the similarity between the text features and the image features of the reference text is less than or equal to the similarity threshold. It should be noted that when there are multiple reference texts, the similarity thresholds corresponding to different reference texts may be the same or different, and this embodiment does not limit the similarity threshold. For another example, the matching condition may include: the category of the reference text is different from the category of the image to be detected.

[0040] For example, the additional information includes at least one of the following: a website, a QR code, a barcode, a mobile phone number, and an account name of a predetermined chat software. Feature extraction can be performed on the reference image containing the additional information to obtain features of the reference image, and the features of the reference image can be used as reference features, or features of multiple reference images can be fused and the fused features can be used as reference features. The number of reference features can be one or more.

[0041] In operation S230, a detection result of the image to be detected is determined according to the similarity between the image feature and each reference feature, wherein the detection result indicates whether the image to be detected includes additional information and, if the additional information is included, indicates the type of the additional information.

[0042] For example, if the similarities between the image feature and each reference feature are less than the corresponding threshold, it can be determined that the detection result is that the image to be detected does not contain additional information. If a target similarity among multiple similarities is greater than the corresponding threshold, it can be determined that the detection result is that the image to be detected contains additional information, and the type of additional information contained in the reference image used to calculate the above target similarity is the type of additional information contained in the image to be detected.

[0043] According to the technical solution provided by the embodiment of the present disclosure, the detection result of the image to be detected can be determined based on multimodal data such as reference text and reference image, so as to effectively identify and classify various forms of additional information used for diversion, and effectively improve the accuracy and robustness of diversion information detection. In addition, the pictures uploaded by users can also be automatically reviewed to improve the review efficiency and reduce the workload of manual review.

[0044] Figure 3is a schematic flowchart of an image detection method according to another embodiment of the present disclosure.

[0045] like Figure 3 As shown, the image detection method 300 may include operation S310, operation S321 to operation S323 and operation S330.

[0046] In operation S310 , image features of an image to be detected are determined.

[0047] In operation S321, it is determined whether the text features and image features of at least one reference text meet the matching condition. If not, operation S322 is performed; if yes, operation S323 is performed.

[0048] In operation S322, it is determined that the detection result of the image to be detected indicates that: the image to be detected includes additional information, and the type of the additional information of the image to be detected is consistent with the type of the target reference text. The above-mentioned target reference text is a reference text whose text features and image features of the image to be detected do not meet the matching condition, for example, the similarity between the text features of the target reference text and the image features of the image to be detected is greater than the similarity threshold. The target reference text can be part or all of the reference text.

[0049] In operation S323, similarities between the image features and each reference feature are determined, wherein the reference feature is determined based on features of the reference image including the additional information, and N is an integer greater than or equal to 1.

[0050] In operation S330, a detection result of the image to be detected is determined according to the similarity between the image feature and each reference feature, wherein the detection result includes: whether the image to be detected includes additional information and the type of the additional information when the additional information is included.

[0051] In this embodiment, when the similarity between the text feature of the reference text and the image feature of the image to be detected is less than or equal to the similarity threshold, the detection result is directly determined. When the similarity is greater than the similarity threshold, the reference feature is used to perform detection again to determine the detection result. This hierarchical detection can improve the detection efficiency and avoid performing the operation of determining the similarity between the image feature and the reference feature for all the image features to be detected, thereby improving the calculation efficiency and ensuring the accuracy of the detection result.

[0052] Figure 4 It is a schematic diagram of the image detection method according to an embodiment of the present disclosure.

[0053] In this embodiment, the deep learning model may be a CLIP model, and the deep learning model includes an image encoder ImageEncoder and a text encoder Text Encoder.

[0054] The image encoder Image Encoder can be used to preprocess and extract features of the image to be detected Image to generate the image feature F_I1 of the image to be detected Image. The image encoder Image Encoder can convert the input image to be detected Image into a high-dimensional feature vector. The image encoder Image Encoder can effectively capture the details and global information in the image, and through the multi-level feature extraction of the neural network, ensure that the generated image feature F_I1 can fully represent the content of the image to be detected.

[0055] Next, the reference text can be determined. For example, the reference text is obtained by combining the template text Text_1 and the type text Text_2, wherein the template text Text_1 is used to describe that the image contains additional information, and the type text Text_2 is used to describe the type of the additional information. For example, the template text Text_1 can be "a picture containing XX", and the type text Text_2 can be: WeChat ID, website, QQ number, QR code, etc. The combined reference text can be "a picture containing WeChat ID" Text_31, "a picture containing a website" Text_32, "a picture containing a QQ number" Text_33, "a picture containing a QR code" Text_34, etc. In this example, the reference text is obtained by combining the template text Text_1 and the type text Text_2, so that the reference text can play a role in guiding the deep learning model, thereby improving the detection accuracy of the deep learning model.

[0056] Next, the reference text can be encoded into text features using a text encoder. This step converts the reference text into text features so that it can be compared with the image feature F_I1 vector. For example, the text feature F_T1 of the reference text Text_31, the text feature F_T2 of the reference text Text_32, the text feature F_T3 of the reference text Text_33, and the text feature F_T4 of the reference text Text_34 are obtained.

[0057] Next, the deep learning model can be used to calculate the similarity between the image feature F_I1 of the image to be detected and the text features of these reference texts, and make a preliminary classification decision. For example, the cosine similarity between the image feature F_I1 of the image to be detected and the text features of the reference text can be calculated by the CLIP model to determine the matching degree between the image to be detected and the reference text. This process can effectively identify whether the image to be detected contains additional information for drainage and make a preliminary classification.

[0058] For example, by calculation, it is possible to determine the similarity I1T1 between the image feature F_I1 of the image to be detected Image and the text feature F_T1 of the reference text Text_31, the similarity I1T2 between the image feature F_I1 of the image to be detected Image and the text feature F_T2 of the reference text Text_32, the similarity I1T3 between the image feature F_I1 of the image to be detected Image and the text feature F_T3 of the reference text Text_33, and the similarity I1T4 between the image feature F_I1 of the image to be detected Image and the text feature F_T4 of the reference text Text_34. If the similarity I1T2 and the similarity I1T4 are greater than the corresponding thresholds, it can be determined that the image to be detected Image includes a website and a QR code.

[0059] Figure 5 is a schematic diagram of an image detection method according to another embodiment of the present disclosure.

[0060] In this embodiment, if the similarities between the image to be detected Image and multiple reference texts are all less than the corresponding threshold, they can be classified by reference features, so as to further confirm whether the image to be detected Image contains additional information and the type of additional information through a more sophisticated classification method.

[0061] For example, a reference feature may be constructed first. For example, at least one reference image set may be obtained, each reference image set including at least one reference image, wherein the reference images in the same reference image set include the same type of additional information, and the types of additional information in different reference image sets are different from each other.

[0062] For example, reference image set Set_1 includes multiple reference images containing WeChat accounts, reference image set Set_2 includes multiple reference images containing URLs, reference image set Set_3 includes multiple reference images containing QQ accounts, and reference image set Set_4 includes multiple reference images containing QR codes.

[0063] Next, for each reference image set in at least one reference image set, the features of the reference images in the reference image set may be fused to obtain reference features. For example, for reference image set Set_1, the features of the reference images in the reference image set Set_1 may be extracted by an image encoder Image Encoder to obtain multiple features, and then the fused features may be obtained by calculating the average value or weighted value of the multiple features or performing clustering or other fusion processing, and the fused features may be used as the reference features F_I of the reference image set Set_1. A1 In this way, each reference image set corresponds to a reference feature, for example, the reference image set Set_2 corresponds to the reference feature F_I A2, the reference image set Set_3 corresponds to the reference feature F_I A3 , the reference image set Set_4 corresponds to the reference feature F_I A4 In this example, reference features are determined by a reference image set rather than by a single reference, so that the reference features can characterize multiple representative reference images, thereby improving classification accuracy.

[0064] Next, the similarity between the image feature F_I1 of the image to be detected and the reference feature can be determined. The similarity can be calculated using the cosine distance. The cosine distance calculation can effectively distinguish different types of feature vectors, thereby improving the accuracy of classification. By comparing the similarity, the classification decision is made again to further confirm whether the image to be detected includes a certain type of additional information. For example, the reference feature F_I A3 If the similarity between the image feature F_I1 and the image feature F_I2 of the image to be detected Image is high, the detection result can be determined as: the image to be detected Image contains additional information, and the type of the additional information is QQ number.

[0065] It should be noted that the image encoder used to extract the features of the reference image and the image encoder used to extract the image features of the image to be detected may be the same image encoder or different image encoders.

[0066] In this embodiment, the deep learning model can effectively classify and identify different types of additional information. The above classification process ensures the detection accuracy, robustness and generalization ability of the deep learning model in practical applications, thereby improving the overall effect of image detection.

[0067] According to another embodiment of the present disclosure, image detection can be performed based on a deep learning model, and the deep learning model is trained using training samples, and the training samples include sample images and sample text groups. At least two sample texts in the same sample text group describe the same sample image from different dimensions. For example, the sample text group includes a first sample text and a second sample text, the first sample text is used to describe the content of the sample image, and the second sample text is used to describe whether the sample image includes additional information and the type of additional information when the additional information is included. It can be seen that of the above two types of sample texts, one describes the image content and the other describes the drainage situation of the image, so that training can be performed from two dimensions, thereby improving the detection accuracy of the deep learning model.

[0068] In some embodiments, some original images may be acquired in advance and used as sample images. The original images may also be subjected to data enhancement processing, such as random cropping, rotation, color jittering, etc., to cope with the diversity and concealment of the drainage information.

[0069] Figure 6 It is a schematic diagram of the deep learning model training method according to an embodiment of the present disclosure.

[0070] In this embodiment, the deep learning model includes an image encoder 602 and a text encoder 601. The image encoder 602, for example, uses a convolutional neural network or is based on ViT (Vision Transformer, a deep learning model based on an attention mechanism), and the text encoder 601, for example, uses BERT (Bidirectional Encoder Representations from Transformers, a language representation model). During the image detection process, the image features of the image to be detected can be determined based on the image encoder 602, and the text features of the reference text can be obtained based on the text encoder 601. Next, the training process of the deep learning model is described.

[0071] For example, at least two sample texts in the sample text group are respectively encoded based on the text encoder 601 to obtain at least two sample text features. The sample image 605 can be encoded based on the image encoder 602 to obtain the sample image feature. Then, multiple loss values ​​can be determined based on the similarity between the sample text features and the sample image features of each sample text in the sample text group, and the matching relationship between the sample text and the sample image 605. Then, based on the multiple loss values, the total loss value is calculated, and the parameters of the deep learning model are adjusted based on the total loss until the deep learning model converges. The multiple sample texts in this example describe the sample image 605 from different dimensions, so that the generalization of the deep learning model can be improved through multiple tasks.

[0072] The training process is described by taking the sample text group including the first sample text 603 and the second sample text 604 as an example.

[0073] The first sample text 603 and the sample image 605 can be constructed as a first data pair, and the first sample text 603 is used to describe the content of the sample image 605. For example, the sample image 605 includes a dog and grass, and the first sample text 603 can be "There is a dog sitting on the grass." If the first sample text 603 matches the sample image 605, it means that the image content described by the first sample text 603 is the content of the sample image 605. At this time, the label of the first data pair can be a first numerical value, and the first numerical value can be 1. If the first sample text 603 does not match the sample image 605, it means that the image content described by the first sample text 603 is not the content of the sample image 605 but the content of other images. At this time, the label of the first data pair can be a second numerical value, and the second numerical value can be 0. For example, the M sample images 605 and the M first sample texts 603 in the training sample are matched one by one, such as the first sample image 605 matches the first first sample text 603, and the second sample image 605 matches the second first sample text 603. Figure 6 The diagonal line from the upper left to the lower right of the interaction matrix on the left side represents the similarity. The first data pairs corresponding to these similarities are positive samples, and the first data pairs corresponding to the remaining similarities are negative samples. M is an integer greater than or equal to 1.

[0074] During the training process, the text encoder 601 can be used to convert the first sample text 603 into the first sample text features A1~Am. At the same time, the image encoder 602 converts the sample image 605 into sample image features C1~Cm. Then the similarity between the first sample text features A1~Am of the first sample text 603 and the sample image features C1~Cm of the sample image 605 is calculated, and the difference between the similarity and the label is calculated by the loss function to obtain the first loss value. The loss function is, for example, InfoNCE. In this way, the deep learning model can distinguish between matching and mismatching image-text pairs, thereby enhancing its cross-modal alignment capability. This embodiment performs comparative learning between the sample image 605 and the first sample text 603, and through the refined comparative learning mechanism between the sample image 605 and the first sample text 603, the deep learning model is prompted to learn the complex correspondence between the image visual content and the natural language description.

[0075] The second sample text 604 and the sample image 605 can be constructed as a second data pair, and the second sample text 604 is used to describe whether the sample image 605 includes additional information and the type of additional information if the additional information is included. For example, the sample image 605 includes a dog and grass, and the second sample text 604 matching the sample image 605 can be "the image does not contain additional information". For another example, the sample image 605 includes a dog, grass, and a phone number, and the second sample text 604 matching the sample image 605 can be "the image contains additional information, the type of additional information is the phone number". If the second sample text 604 matches the sample image 605, it means that the image content described by the second sample text 604 is the sample image 605. If the second sample text 604 does not match the sample image 605, it means that the image content described by the second sample text 604 is not the content of the sample image 605 but the content of other images. The label of the second data pair can indicate whether the sample image 605 matches the second sample text 604, and the label of the second data pair can also indicate whether the sample image 605 contains additional information and the type of additional information if the additional information is included. For example, the M sample images 605 and the M second sample texts 604 in the training sample are matched one by one, such as the first sample image 605 and the first second sample text 604 are matched, and the second sample image 605 and the second second sample text 604 are matched. In this way, in the calculated interaction matrix, Figure 6 The diagonal line from the upper left to the lower right of the interaction matrix on the middle right side represents the similarities. The second data pairs corresponding to these similarities are positive samples, and the second data pairs corresponding to the remaining similarities are negative samples.

[0076] During the training process, the text encoder 601 can be used to convert the second sample text 604 into second sample text features B1~Bm. Then, the similarity between the second sample text features B1~Bm of the second sample text 604 and the sample image features C1~Cm of the sample image 605 is calculated, and the difference between the similarity and the label is calculated by the loss function to obtain a second loss value. This embodiment continuously iterates and optimizes its image encoder 602 through a comparative learning framework, so that the image features generated by it can more effectively reflect the essential characteristics of the additional information. At the same time, the label indicating the type of additional information is used as a supervisory signal, and the deep learning model is guided by the interaction matrix to map these features to the correct classification label, thereby achieving high-precision classification prediction.

[0077] After obtaining the first loss value and the second loss value, the first loss value and the second loss value can be weighted to obtain the total loss value, and the parameters of the deep learning model can be adjusted based on the total loss until the deep learning model converges. Based on the first loss value and the second loss value, the deep learning model can not only understand the relationship between the image and the text description, but also effectively classify different additional information in the image. The above two loss values ​​ensure the detection accuracy and generalization ability of the deep learning model in practical applications from two dimensions.

[0078] The above is an explanation of the training process of the deep learning model. The deep learning model trained by the above method has high recognition accuracy and strong generalization ability. It can recognize additional information in the image to be detected and determine the type of additional information. The trained deep learning model can more comprehensively understand and utilize multimodal information by processing image and text data at the same time, thereby improving the effect of diversion image detection.

[0079] Figure 7 It is a schematic structural block diagram of an image detection device according to an embodiment of the present disclosure.

[0080] like Figure 7 As shown, the image detection device 700 may include an image feature determination module 710 , a similarity determination module 720 and a first result determination module 730 .

[0081] The image feature determination module 710 is used to determine the image features of the image to be detected.

[0082] The similarity determination module 720 is used to determine the similarity between the image feature and each reference feature in response to detecting that the text feature and the image feature of at least one reference text meet the matching condition; wherein the reference feature is determined based on the feature of the reference image including additional information.

[0083] The first result determination module 730 is used to determine the detection result of the image to be detected according to the similarity between the image feature and each reference feature, wherein the detection result indicates whether the image to be detected includes additional information and the type of the additional information if the additional information is included.

[0084] According to another embodiment of the present disclosure, the reference feature is determined by the following modules: an acquisition module and a fusion module. The acquisition module is used to acquire at least one reference image set, each reference image set including at least one reference image; wherein the reference images in the same reference image set include the same type of additional information, and the types of additional information in different reference image sets are different from each other. The fusion module is used to fuse the features of the reference images in the reference image set for each reference image set in at least one reference image set to obtain a reference feature; wherein the reference feature corresponds to the reference image set one by one.

[0085] According to another embodiment of the present disclosure, the reference text is obtained by combining a type text and a template text, wherein the type text is used to describe the type of the additional information, and the template text is used to describe that the image contains the additional information.

[0086] According to another embodiment of the present disclosure, it also includes: a second result determination module, which is used to determine a detection result in response to detecting that the text features and image features of at least part of the at least one reference text do not meet the matching conditions, and the detection result indicates that: the image to be detected includes additional information, and the type of the additional information of the image to be detected is consistent with the type of at least part of the reference text.

[0087] According to another embodiment of the present disclosure, the additional information includes at least one of the following: a website, a QR code, a barcode, a mobile phone number, and an account name of a predetermined chat software.

[0088] According to another embodiment of the present disclosure, the matching condition includes: the similarity between the text feature and the image feature of the reference text is not greater than a similarity threshold.

[0089] According to another embodiment of the present disclosure, the image features of the image to be detected are obtained based on the image encoder, and the text features of the reference text are obtained based on the text encoder; the image encoder and the text encoder are obtained by training using training samples, and the training samples include sample images and sample text groups, and at least two sample texts in the same sample text group describe the same sample image from different dimensions; the image encoder and the text encoder are obtained by training through the following modules: a first encoding module, a second encoding module, a loss determination module, and a training module. The first encoding module is used to encode at least two sample texts in the sample text group based on the text encoder, respectively, to obtain at least two sample text features. The second encoding module is used to encode the sample image based on the image encoder to obtain the sample image feature. The loss determination module is used to determine multiple loss values ​​based on the similarity between the sample text feature and the sample image feature of each sample text in the sample text group, and the matching relationship between the sample text and the sample image. The training module is used to train the image encoder and the text encoder based on the multiple loss values.

[0090] According to another embodiment of the present disclosure, the sample text group includes a first sample text and a second sample text, wherein the first sample text is used to describe the content of the sample image, and the second sample text is used to describe whether the sample image includes additional information and the type of the additional information if the additional information is included.

[0091] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned image detection method.

[0092] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above-mentioned image detection method.

[0093] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, and the computer program implements the above-mentioned image detection method when executed by a processor.

[0094] Figure 8 1 is a block diagram of an electronic device for implementing the image detection method of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0095] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0096] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0097] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as image detection methods. For example, in some embodiments, the image detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the image detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the image detection method in any other appropriate manner (e.g., by means of firmware).

[0098] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0102] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0103] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image detection method, comprising: Determine image features of the image to be detected; In response to detecting that both the text feature of at least one reference text and the image feature satisfy a matching condition, determining a similarity between the image feature and each reference feature; wherein the reference feature is determined based on a feature of the reference image including additional information; and A detection result of the image to be detected is determined according to the similarity between the image feature and each reference feature, wherein the detection result indicates whether the image to be detected includes additional information and the type of the additional information if the additional information is included.

2. The method according to claim 1, wherein: The reference characteristics are determined by: Acquire at least one reference image set, each reference image set including at least one reference image; wherein the reference images in the same reference image set include the same type of additional information, and the types of the additional information in different reference image sets are different from each other; and For each reference image set in the at least one reference image set, features of the reference images in the reference image set are fused to obtain the reference features; wherein the reference features correspond to the reference image sets one by one.

3. The method according to claim 1, wherein: The reference text is obtained by combining a type text and a template text, wherein the type text is used to describe the type of the additional information, and the template text is used to describe that the image contains the additional information.

4. The method according to claim 1, further comprising: In response to detecting that text features of at least part of the at least one reference text and the image features do not satisfy the matching condition, the detection result is determined, wherein the detection result indicates that: the image to be detected includes additional information, and the type of the additional information of the image to be detected is consistent with the type of the at least part of the reference text.

5. The method according to claim 1, wherein: The additional information includes at least one of the following: a website, a QR code, a barcode, a mobile phone number, and an account name of a predetermined chat software.

6. The method according to claim 1, wherein: The matching condition includes: the similarity between the text feature of the reference text and the image feature is not greater than a similarity threshold.

7. The method according to any one of claims 1 to 6, wherein: The image feature of the image to be detected is obtained based on an image encoder, and the text feature of the reference text is obtained based on a text encoder; the image encoder and the text encoder are trained using training samples, the training samples include sample images and sample text groups, and at least two sample texts in the same sample text group describe the same sample image from different dimensions; the image encoder and the text encoder are trained in the following manner: Encode at least two sample texts in the sample text group based on the text encoder to obtain at least two sample text features; Encoding the sample image based on the image encoder to obtain sample image features; Determining a plurality of loss values ​​according to the similarity between the sample text feature of each sample text in the sample text group and the sample image feature, and the matching relationship between the sample text and the sample image; and The image encoder and the text encoder are trained according to the multiple loss values.

8. The method according to claim 7, wherein: The sample text group includes a first sample text and a second sample text, wherein the first sample text is used to describe the content of the sample image, and the second sample text is used to describe whether the sample image includes additional information and the type of the additional information when the additional information is included.

9. An image detection device, comprising: An image feature determination module, used to determine the image features of the image to be detected; a similarity determination module, configured to determine the similarity between the image feature and each reference feature in response to detecting that both the text feature of at least one reference text and the image feature satisfy a matching condition; wherein the reference feature is determined based on the feature of the reference image including the additional information; and The first result determination module is used to determine the detection result of the image to be detected based on the similarity between the image feature and the respective reference features, wherein the detection result indicates whether the image to be detected includes additional information and the type of additional information when the additional information is included.

10. The device according to claim 9, wherein: The reference characteristics are determined by the following modules: an acquisition module, configured to acquire at least one reference image set, each reference image set comprising at least one reference image; wherein the reference images in the same reference image set comprise additional information of the same type, and the types of the additional information in different reference image sets are different from each other; and A fusion module is used to fuse the features of the reference images in each reference image set in the at least one reference image set to obtain the reference features; wherein the reference features correspond to the reference image sets one by one.

11. The device according to claim 9, wherein: The reference text is obtained by combining a type text and a template text, wherein the type text is used to describe the type of the additional information, and the template text is used to describe that the image contains the additional information.

12. The apparatus according to claim 9, further comprising: The second result determination module is used to determine the detection result in response to detecting that the text features of at least part of the at least one reference text and the image features do not meet the matching condition, and the detection result indicates that: the image to be detected includes additional information, and the type of the additional information of the image to be detected is consistent with the type of the at least part of the reference text.

13. The device according to claim 9, wherein: The additional information includes at least one of the following: a website, a QR code, a barcode, a mobile phone number, and an account name of a predetermined chat software.

14. The device according to claim 9, wherein: The matching condition includes: the similarity between the text feature of the reference text and the image feature is not greater than a similarity threshold.

15. The device according to any one of claims 9 to 14, wherein: The image feature of the image to be detected is obtained based on an image encoder, and the text feature of the reference text is obtained based on a text encoder; the image encoder and the text encoder are trained using training samples, the training samples include sample images and sample text groups, and at least two sample texts in the same sample text group describe the same sample image from different dimensions; the image encoder and the text encoder are trained using the following modules: A first encoding module, used for encoding at least two sample texts in the sample text group respectively based on the text encoder to obtain at least two sample text features; A second encoding module, used for encoding the sample image based on the image encoder to obtain sample image features; A loss determination module, used for determining a plurality of loss values ​​according to the similarity between the sample text feature of each sample text in the sample text group and the sample image feature, and the matching relationship between the sample text and the sample image; as well as A training module is used to train the image encoder and the text encoder according to the multiple loss values.

16. The device according to claim 15, wherein: The sample text group includes a first sample text and a second sample text, wherein the first sample text is used to describe the content of the sample image, and the second sample text is used to describe whether the sample image includes additional information and the type of the additional information when the additional information is included.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images