Image detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image detection method and device, electronic equipment and storage medium, relates to the technical field of artificial intelligence, in particular to the fields of deep learning, image processing and the like. The specific implementation scheme is as follows: determining an image feature of a to-be-detected image; in response to detecting that the text feature of at least one reference text and the image feature both satisfy a matching condition, determining the similarity between the image feature and each reference feature; wherein the reference feature is determined according to the feature of a reference image including additional information; and determining a detection result of the to-be-detected image according to the similarity between the image feature and each reference feature, wherein the detection result indicates whether the to-be-detected image includes additional information and, in the case of including additional information, the type of the additional information.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of deep learning and image processing. More specifically, the present disclosure provides an image detection method, device, electronic device, storage medium, and computer program product. Background Art

[0002] Some users insert additional information such as QR codes and phone numbers into images to divert traffic. This additional information not only violates the platform's user rules but can also be easily exploited by illicit actors for online fraud and illegal traffic diversion. Therefore, image review is necessary to maintain platform rules and improve security. Summary of the Invention

[0003] The present disclosure provides an image detection method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to one aspect of the present disclosure, an image detection method is provided, comprising: determining image features of an image to be detected; in response to detecting that both text features and image features of at least one reference text satisfy matching conditions, determining the similarity between the image features and each reference feature; wherein the reference features are determined based on features of a reference image including additional information; and determining a detection result of the image to be detected based on the similarity between the image features and each reference feature, wherein the detection result indicates: whether the image to be detected includes additional information, and, if additional information is included, the type of the additional information.

[0005] According to another aspect of the present disclosure, an image detection device is provided, comprising: an image feature determination module, a similarity determination module, and a first result determination module. The image feature determination module is used to determine the image features of an image to be detected. The similarity determination module is used to determine the similarity between the image features and each reference feature in response to detecting that both the text features and the image features of at least one reference text satisfy a matching condition; wherein the reference features are determined based on the features of a reference image including additional information. The first result determination module is used to determine a detection result of the image to be detected based on the similarity between the image features and each reference feature, wherein the detection result indicates: whether the image to be detected includes additional information, and the type of additional information if additional information is included.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided in the present disclosure when executed by a processor.

[0009] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an application scenario of the image detection method and device according to an embodiment of the present disclosure;

[0012] Figure 2 is a schematic flow chart of an image detection method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic flow chart of an image detection method according to another embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of an image detection method according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of an image detection method according to another embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram of a deep learning model training method according to an embodiment of the present disclosure;

[0017] Figure 7 is a schematic structural block diagram of an image detection device according to an embodiment of the present disclosure; and

[0018] Figure 8 It is a structural block diagram of an electronic device used to implement the image detection method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0021] In the technical solution disclosed herein, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0022] Related technologies include OCR (Optical Character Recognition) or QR code detection to detect whether an image contains additional information used for traffic diversion. However, these solutions lack robustness and generalizability. When the additional information used for traffic diversion is hidden in an image through methods such as artistic fonts, homophones, emoticons, AI-powered QR codes, and handwriting, detection failures are common.

[0023] The disclosed embodiment provides an image detection method, which first determines whether the image features of the image to be detected and the text features of the reference text meet the matching conditions, and then determines the detection result of the image to be detected based on the similarity between the image features and the reference features, and the above-mentioned reference features are determined based on the features of the reference image including additional information. By adopting the above-mentioned technical solution, the detection result of the image to be detected can be determined based on multimodal data such as reference text and reference image, thereby effectively identifying and classifying various forms of additional information for diversion, and effectively improving the accuracy and robustness of diversion information detection. In addition, it is also possible to automatically review the pictures uploaded by users, improve the review efficiency, and reduce the workload of manual review.

[0024] The image detection method provided by the embodiments of the present disclosure can be applied to a variety of products or projects that require content review and traffic diversion detection of images, thereby improving the platform's ability to detect cheating users and ensuring a healthy platform ecosystem. For example, it can be applied to social media platforms to detect additional information in user-uploaded images to prevent inappropriate promotion and traffic diversion. Another example is that it can be applied to e-commerce platforms to review product images uploaded by sellers, preventing merchants from using images to divert users to other platforms for transactions and protecting users from being deceived by unscrupulous merchants. Another example is that it can be applied to advertising review systems for advertising companies or platforms to review the content of advertising images to ensure legality and compliance. Another example is that it can be applied to content review systems such as news websites and forums to review the content of user-uploaded images and prevent malicious traffic diversion through images. Another example is that it can be applied to network security systems for various network security companies or platforms to detect and prevent malicious information spread through images, thereby improving network security. Another example is that it can be applied to education and training, in educational platforms, training institutions, and other occasions, to prevent the spread of harmful information through images.

[0025] The technical solutions provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Figure 1 Schematic diagram of an application scenario of the image detection method and device according to an embodiment of the present disclosure.

[0027] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be used in other devices, systems, environments or scenarios.

[0028] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers, etc.

[0030] Server 105 may be a server that provides various services, such as a backend management server (for example only) that supports websites browsed by users using terminal devices 101, 102, and 103. The backend management server may analyze and process received data such as user requests, and provide feedback to the terminal device regarding the processing results (e.g., determining a detection result based on an image uploaded by a user and, based on the detection result, determining whether to display the image to other users or block the image).

[0031] It should be noted that the image detection method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image detection device provided in the embodiment of the present disclosure can generally be set in the server 105. The image detection method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image detection device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0033] Figure 2 is a schematic flow chart of an image detection method according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, the image detection method 200 may include operations S210 to S230.

[0035] In operation S210 , image features of an image to be detected are determined.

[0036] For example, a deep learning model can be pre-trained and used to extract image features from the image to be detected. For example, the deep learning model includes an image encoder, which is used to extract image features from the image to be detected. For example, the deep learning model can be a CLIP (Contrastive Language–Image Pre-training) model.

[0037] In operation S220, in response to detecting that both text features and image features of at least one reference text satisfy a matching condition, similarities between the image features and respective reference features are determined; wherein the reference features are determined based on features of the reference image including additional information.

[0038] For example, the reference text can be text related to the purpose of traffic diversion. For example, the reference text is a character describing information used for traffic diversion, such as a mobile phone number or a website address. Another example is a text describing an image used for traffic diversion, such as "a picture containing a mobile phone number" or "a picture containing a QR code." Another example is a text describing the patterns or characteristics of information used for traffic diversion, such as examples describing the number of digits or the first few digits. For example, a deep learning model can also include a text encoder, which can be used to extract text features of the reference text.

[0039] For example, the matching condition may include: the similarity between the text features of the reference text and the image features is less than or equal to a similarity threshold. It should be noted that when there are multiple reference texts, the similarity thresholds corresponding to different reference texts may be the same or different, and this embodiment does not limit the similarity threshold. For another example, the matching condition may include: the category of the reference text is different from the category of the image to be detected.

[0040] For example, the additional information includes at least one of the following: a website, a QR code, a barcode, a mobile phone number, and a pre-determined chat software account name. Feature extraction can be performed on the reference image containing the additional information to obtain reference image features. The reference image features can be used as reference features, or features of multiple reference images can be fused and the fused features used as reference features. The number of reference features can be one or more.

[0041] In operation S230 , a detection result of the image to be detected is determined based on the similarity between the image feature and each reference feature, wherein the detection result indicates whether the image to be detected includes additional information and, if so, the type of the additional information.

[0042] For example, if the similarities between the image feature and each reference feature are all less than corresponding thresholds, the detection result can be determined to be that the image to be detected does not contain additional information. If a target similarity among multiple similarities is greater than a corresponding threshold, the detection result can be determined to be that the image to be detected contains additional information, and the type of additional information contained in the reference image used to calculate the target similarity is the same as the type of additional information contained in the image to be detected.

[0043] The technical solutions provided by the embodiments of this disclosure can determine the detection results of the image to be detected based on multimodal data such as reference text and reference images, effectively identifying and classifying various forms of additional information used for traffic diversion, and effectively improving the accuracy and robustness of traffic diversion information detection. Furthermore, it can also automatically review user-uploaded images, improving review efficiency and reducing the workload of manual review.

[0044] Figure 3is a schematic flowchart of an image detection method according to another embodiment of the present disclosure.

[0045] like Figure 3 As shown, the image detection method 300 may include operation S310, operations S321 to S323, and operation S330.

[0046] In operation S310 , image features of an image to be detected are determined.

[0047] In operation S321, it is determined whether the text features and image features of at least one reference text meet a matching condition. If not, operation S322 is performed; if so, operation S323 is performed.

[0048] In operation S322, it is determined whether the detection result of the image to be detected indicates that the image to be detected includes additional information, and the type of the additional information of the image to be detected is consistent with the type of the target reference text. The target reference text is a reference text whose text features do not meet a matching condition with the image features of the image to be detected, for example, the similarity between the text features of the target reference text and the image features of the image to be detected is greater than a similarity threshold. The target reference text may be part or all of the reference text.

[0049] In operation S323 , similarities between the image features and respective reference features are determined, where the reference features are determined based on features of the reference image including the additional information, and N is an integer greater than or equal to 1.

[0050] In operation S330 , a detection result of the image to be detected is determined based on the similarity between the image feature and each reference feature, wherein the detection result includes whether the image to be detected includes additional information and the type of the additional information if the additional information is included.

[0051] In this embodiment, when the similarity between the text features of the reference text and the image features of the image to be detected is less than or equal to the similarity threshold, the detection result is directly determined. When the similarity is greater than the similarity threshold, the reference features are used for further detection to determine the detection result. This hierarchical detection method can improve detection efficiency and avoid performing the operation of determining the similarity between the image features and the reference features for all image features to be detected, thereby improving computational efficiency and ensuring the accuracy of the detection results.

[0052] Figure 4 It is a schematic diagram of the image detection method according to an embodiment of the present disclosure.

[0053] In this embodiment, the deep learning model may be a CLIP model, and the deep learning model includes an image encoder ImageEncoder and a text encoder Text Encoder.

[0054] An image encoder can be used to preprocess and extract features from the image to be detected, generating image features F_I1. The image encoder converts the input image to be detected into a high-dimensional feature vector. The image encoder effectively captures both detailed and global information within the image. Through multi-layered feature extraction using a neural network, the generated image features F_I1 fully represent the content of the image to be detected.

[0055] Next, the reference text can be determined. For example, the reference text is obtained by combining the template text Text_1 and the type text Text_2, where the template text Text_1 is used to describe that the image contains additional information, and the type text Text_2 is used to describe the type of additional information. For example, the template text Text_1 can be "a picture containing XX", and the type text Text_2 can be: WeChat ID, website address, QQ number, QR code, etc. The combined reference text can be "a picture containing a WeChat ID" Text_31, "a picture containing a website address" Text_32, "a picture containing a QQ number" Text_33, "a picture containing a QR code" Text_34, etc. In this example, the reference text is obtained by combining the template text Text_1 and the type text Text_2. In this way, the reference text can guide the deep learning model, thereby improving the detection accuracy of the deep learning model.

[0056] Next, we can use a text encoder to encode the reference text into text features. This step converts the reference text into text features so that they can be compared with the image feature vector F_I1. For example, we can obtain the text feature F_T1 of the reference text Text_31, the text feature F_T2 of the reference text Text_32, the text feature F_T3 of the reference text Text_33, and the text feature F_T4 of the reference text Text_34.

[0057] Next, a deep learning model can be used to calculate the similarity between the generated image features F_I1 of the image to be tested and the text features of these reference texts to make a preliminary classification decision. For example, the CLIP model can be used to calculate the cosine similarity between the image features F_I1 of the image to be tested and the text features of the reference text to determine the degree of match between the image to be tested and the reference text. This process can effectively identify whether the image to be tested contains additional information for attracting traffic and perform preliminary classification.

[0058] For example, calculations can be performed to determine the similarity I1T1 between the image feature F_I1 of the image to be detected and the text feature F_T1 of the reference text Text_31; the similarity I1T2 between the image feature F_I1 of the image to be detected and the text feature F_T2 of the reference text Text_32; the similarity I1T3 between the image feature F_I1 of the image to be detected and the text feature F_T3 of the reference text Text_33; and the similarity I1T4 between the image feature F_I1 of the image to be detected and the text feature F_T4 of the reference text Text_34. If similarities I1T2 and I1T4 are greater than corresponding thresholds, it can be determined that the image to be detected includes a URL and a QR code.

[0059] Figure 5 It is a schematic diagram of an image detection method according to another embodiment of the present disclosure.

[0060] In this embodiment, if the similarities between the image to be detected Image and multiple reference texts are all less than the corresponding threshold, they can be classified by reference features, so as to further confirm whether the image to be detected Image contains additional information and the type of additional information through a more refined classification method.

[0061] For example, a reference feature may be constructed first. For example, at least one reference image set may be obtained, each reference image set including at least one reference image, wherein the reference images in the same reference image set include the same type of additional information, and the types of additional information in different reference image sets are different.

[0062] For example, reference image set Set_1 includes multiple reference images containing WeChat accounts, reference image set Set_2 includes multiple reference images containing URLs, reference image set Set_3 includes multiple reference images containing QQ accounts, and reference image set Set_4 includes multiple reference images containing QR codes.

[0063] Next, for each reference image set in at least one reference image set, the features of the reference images in the reference image set can be fused to obtain reference features. For example, for reference image set Set_1, the features of the reference images in the reference image set Set_1 can be extracted by an image encoder Image Encoder to obtain multiple features. Then, the fused features can be obtained by calculating the average or weighted value of the multiple features or performing clustering or other fusion processing. The fused features can be used as the reference features F_I of the reference image set Set_1. A1 In this way, each reference image set corresponds to a reference feature, for example, the reference image set Set_2 corresponds to the reference feature F_I A2, the reference image set Set_3 corresponds to the reference feature F_I A3 , the reference image set Set_4 corresponds to the reference feature F_I A4 This example determines reference features through a reference image set rather than a single reference, so that the reference features can characterize multiple representative reference images, thereby improving classification accuracy.

[0064] Next, the similarity between the image feature F_I1 of the image to be detected and the reference feature can be determined. The similarity can be calculated using the cosine distance. The cosine distance calculation can effectively distinguish different types of feature vectors, thereby improving the accuracy of classification. By comparing the similarity, the classification decision is made again to further confirm whether the image to be detected includes a certain type of additional information. For example, the reference feature F_I A3 If the similarity between the image feature F_I1 and the image feature F_I2 of the image to be detected is high, it can be determined that the detection result is: the image to be detected Image contains additional information, and the type of the additional information is QQ number.

[0065] It should be noted that the image encoder used to extract features of the reference image and the image encoder used to extract features of the image to be detected may be the same image encoder or different image encoders.

[0066] In this embodiment, the deep learning model can effectively classify and identify different types of additional information. The above classification process ensures the detection accuracy, robustness and generalization ability of the deep learning model in practical applications, thereby improving the overall effect of image detection.

[0067] According to another embodiment of the present disclosure, image detection can be performed based on a deep learning model, and the deep learning model is obtained by training using training samples, and the training samples include sample images and sample text groups. At least two sample texts in the same sample text group describe the same sample image from different dimensions. For example, the sample text group includes a first sample text and a second sample text, the first sample text is used to describe the content of the sample image, and the second sample text is used to describe whether the sample image includes additional information and the type of additional information when the additional information is included. It can be seen that of the above two sample texts, one describes the image content and the other describes the image drainage situation, so that training can be performed from two dimensions, thereby improving the detection accuracy of the deep learning model.

[0068] In some embodiments, some original images can be pre-acquired and used as sample images. Data augmentation processing can also be performed on the original images, such as random cropping, rotation, and color dithering, to address the diversity and concealment of the traffic diversion information.

[0069] Figure 6 It is a schematic diagram of the deep learning model training method according to an embodiment of the present disclosure.

[0070] In this embodiment, the deep learning model includes an image encoder 602 and a text encoder 601. The image encoder 602, for example, utilizes a convolutional neural network or is based on ViT (Vision Transformer, a deep learning model based on an attention mechanism), while the text encoder 601, for example, utilizes BERT (Bidirectional Encoder Representations from Transformers, a language representation model). During image detection, the image features of the image to be detected can be determined based on the image encoder 602, while the text features of the reference text can be obtained based on the text encoder 601. Next, the training process of the deep learning model is described.

[0071] For example, at least two sample texts in the sample text group are encoded separately based on the text encoder 601 to obtain at least two sample text features. The sample image 605 can be encoded based on the image encoder 602 to obtain sample image features. Then, multiple loss values ​​can be determined based on the similarity between the sample text features and the sample image features of each sample text in the sample text group, as well as the matching relationship between the sample text and the sample image 605. Then, based on the multiple loss values, the total loss value is calculated, and the parameters of the deep learning model are adjusted based on the total loss until the deep learning model converges. The multiple sample texts in this example describe the sample image 605 from different dimensions, so that the generalization of the deep learning model can be improved through multi-tasking.

[0072] The training process is described by taking the sample text group including the first sample text 603 and the second sample text 604 as an example.

[0073] The first sample text 603 and the sample image 605 can be constructed as a first data pair, and the first sample text 603 is used to describe the content of the sample image 605. For example, the sample image 605 includes a dog and grass, and the first sample text 603 can be "There is a dog sitting on the grass." If the first sample text 603 matches the sample image 605, it means that the image content described by the first sample text 603 is the content of the sample image 605. At this time, the label of the first data pair can be a first numerical value, and the first numerical value can be 1. If the first sample text 603 does not match the sample image 605, it means that the image content described by the first sample text 603 is not the content of the sample image 605 but the content of other images. At this time, the label of the first data pair can be a second numerical value, and the second numerical value can be 0. For example, the M sample images 605 and the M first sample texts 603 in the training sample are matched one by one, such as the first sample image 605 matches the first first sample text 603, and the second sample image 605 matches the second first sample text 603. Figure 6 The diagonal line from the upper left to the lower right of the interaction matrix on the left side represents the similarity. The first data pairs corresponding to these similarities are positive samples, and the first data pairs corresponding to the remaining similarities are negative samples. M is an integer greater than or equal to 1.

[0074] During the training process, the text encoder 601 can be used to convert the first sample text 603 into first sample text features A1~Am. At the same time, the image encoder 602 converts the sample image 605 into sample image features C1~Cm. Then the similarity between the first sample text features A1~Am of the first sample text 603 and the sample image features C1~Cm of the sample image 605 is calculated, and the difference between the similarity and the label is calculated by the loss function to obtain the first loss value. The loss function is, for example, InfoNCE. In this way, the deep learning model can distinguish between matching and mismatching image-text pairs, thereby enhancing its cross-modal alignment capability. This embodiment performs comparative learning between the sample image 605 and the first sample text 603, and through the refined comparative learning mechanism between the sample image 605 and the first sample text 603, the deep learning model is prompted to learn the complex correspondence between the image visual content and the natural language description.

[0075] Second sample text 604 and sample image 605 can be constructed into a second data pair. Second sample text 604 is used to describe whether sample image 605 includes additional information and, if so, the type of additional information. For example, if sample image 605 includes a dog and grass, the second sample text 604 matching this sample image 605 may be "The image does not contain additional information." For another example, if sample image 605 includes a dog, grass, and a phone number, the second sample text 604 matching this sample image 605 may be "The image contains additional information, the type of additional information is a phone number." If second sample text 604 matches sample image 605, it indicates that the image content described by second sample text 604 is sample image 605. If second sample text 604 does not match sample image 605, it indicates that the image content described by second sample text 604 is not the content of sample image 605 but the content of another image. The label of the second data pair can indicate whether sample image 605 matches second sample text 604. The label of the second data pair can also indicate whether sample image 605 includes additional information and, if so, the type of additional information. For example, the M sample images 605 and the M second sample texts 604 in the training sample are matched one by one, such as the first sample image 605 and the first second sample text 604 are matched, and the second sample image 605 and the second second sample text 604 are matched. In the interaction matrix calculated in this way, Figure 6 The diagonal line from the upper left to the lower right of the interaction matrix on the middle right side represents similarity. The second data pairs corresponding to these similarities are positive samples, and the second data pairs corresponding to the remaining similarities are negative samples.

[0076] During the training process, the text encoder 601 can be used to convert the second sample text 604 into second sample text features B1~Bm. Then, the similarity between the second sample text features B1~Bm of the second sample text 604 and the sample image features C1~Cm of the sample image 605 is calculated, and the difference between the similarity and the label is calculated by the loss function to obtain a second loss value. This embodiment continuously iteratively optimizes its image encoder 602 through a comparative learning framework, so that the image features generated by it can more effectively reflect the essential characteristics of the additional information. At the same time, the label indicating the type of additional information serves as a supervisory signal, and the deep learning model is guided by the interaction matrix to map these features to the correct classification label, thereby achieving high-precision classification prediction.

[0077] After the first loss value and the second loss value are obtained, the first loss value and the second loss value can be weighted to obtain a total loss value, and the parameters of the deep learning model are adjusted based on the total loss until the deep learning model converges. Based on the training of the first loss value and the second loss value, the deep learning model can not only understand the relationship between the image and the text description, but also effectively classify different additional information in the image. The above two loss values ensure the detection accuracy and generalization ability of the deep learning model in practical application from two dimensions.

[0078] The training process of the deep learning model is described above. The deep learning model trained by the above method has high recognition accuracy and strong generalization ability, can identify additional information in the to-be-detected image, and determine the type of the additional information. The trained deep learning model can more comprehensively understand and utilize multi-modal information by simultaneously processing image and text data, and improve the effect of drainage picture detection.

[0079] Figure 7 is a schematic structural block diagram of an image detection apparatus according to an embodiment of the present disclosure.

[0080] As shown in Figure 7 , the image detection apparatus 700 can include an image feature determination module 710, a similarity determination module 720, and a first result determination module 730.

[0081] The image feature determination module 710 is configured to determine the image feature of the to-be-detected image.

[0082] The similarity determination module 720 is configured to, in response to detecting that the text feature of the at least one reference text and the image feature both satisfy a matching condition, determine the similarity between the image feature and each reference feature, wherein the reference feature is determined according to the feature of a reference image including additional information.

[0083] The first result determination module 730 is configured to determine the detection result of the to-be-detected image according to the similarity between the image feature and each reference feature, wherein the detection result indicates whether the to-be-detected image includes additional information, and in the case of including additional information, the type of the additional information.

[0084] According to another embodiment of the present disclosure, the reference feature is determined by an acquisition module and a fusion module. The acquisition module is configured to acquire at least one reference image set, each reference image set including at least one reference image; wherein the reference images in the same reference image set include the same type of additional information, and the types of additional information of different reference image sets are different from each other. The fusion module is configured to fuse the features of the reference images in each of the at least one reference image set to obtain a reference feature; wherein the reference feature corresponds to the reference image set one by one.

[0085] According to another embodiment of the present disclosure, the reference text is obtained by combining a type text and a template text, wherein the type text is used to describe the type of the additional information, and the template text is used to describe that the image contains the additional information.

[0086] According to another embodiment of the present disclosure, the method further includes: a second result determination module configured to, in response to detecting that the text feature of at least part of the reference text in the at least one reference text and the image feature do not satisfy a matching condition, determine a detection result, the detection result indicating that the image to be detected includes additional information, and the type of the additional information of the image to be detected is consistent with the type of the at least part of the reference text.

[0087] According to another embodiment of the present disclosure, the additional information includes at least one of the following: a website address, a two-dimensional code, a bar code, a mobile phone number, and an account name of a predetermined chat software.

[0088] According to another embodiment of the present disclosure, the matching condition includes that the similarity between the text feature of the reference text and the image feature is not greater than a similarity threshold.

[0089] According to another embodiment of the present disclosure, the image feature of the image to be detected is obtained based on an image encoder, and the text feature of the reference text is obtained based on a text encoder; the image encoder and the text encoder are trained by using training samples, the training samples include sample image groups and sample text groups, at least two sample texts in the same sample text group describe the same sample image from different dimensions; the image encoder and the text encoder are trained by using the following modules: a first encoding module, a second encoding module, a loss determination module, and a training module. The first encoding module is configured to encode at least two sample texts in the sample text group based on the text encoder to obtain at least two sample text features. The second encoding module is configured to encode the sample image based on the image encoder to obtain a sample image feature. The loss determination module is configured to determine a plurality of loss values according to the similarity between the sample text feature of each sample text in the sample text group and the sample image feature, and the matching relationship between the sample text and the sample image. The training module is configured to train the image encoder and the text encoder according to the plurality of loss values.

[0090] According to another embodiment of the present disclosure, the sample text set includes a first sample text and a second sample text, the first sample text is used to describe the content of the sample image, and the second sample text is used to describe whether the sample image includes additional information and the type of the additional information in the case of including the additional information.

[0091] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, including at least one processor; and a memory connected with the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image detection method.

[0092] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to perform the image detection method.

[0093] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, and the computer program is executed by a processor to implement the image detection method.

[0094] Figure 8 is a structural block diagram of an electronic device for implementing the image detection method of the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections, and their functions, as well as their relationships with one another, are merely examples and are not intended to limit the implementations of the present disclosure described and / or claimed herein.

[0095] As shown in Figure 8 , the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0096] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0097] The computing unit 801 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the image detection method. For example, in some embodiments, the image detection method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the image detection method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the image detection method by any other appropriate means, such as by means of firmware.

[0098] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0099] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0102] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0103] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0104] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0105] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An image detection method, comprising: Determining image features of the image to be detected; In response to detecting that both text features of at least one reference text and the image features satisfy a matching condition, determining a similarity between the image features and each reference feature; wherein the reference features are determined based on features of the reference image including the additional information; and determining a detection result of the image to be detected based on the similarity between the image feature and each reference feature, wherein the detection result indicates: whether the image to be detected includes additional information, and the type of the additional information if the additional information is included; The reference text is obtained by combining a type text and a template text, wherein the type text is used to describe the type of additional information and the template text is used to describe that the image contains additional information; the matching condition includes: the similarity between the text features of the reference text and the image features is not greater than a similarity threshold.

2. The method according to claim 1, wherein The reference characteristics are determined by: Acquire at least one reference image set, each reference image set including at least one reference image; wherein the reference images in the same reference image set include the same type of additional information, and the types of the additional information in different reference image sets are different from each other; and For each reference image set in the at least one reference image set, features of the reference images in the reference image set are fused to obtain the reference features; wherein the reference features correspond to the reference image sets one-to-one.

3. The method according to claim 1, further comprising: In response to detecting that the text features of at least part of the reference text in the at least one reference text do not meet the matching condition with the image features, the detection result is determined, wherein the detection result indicates that: the image to be detected includes additional information, and the type of the additional information of the image to be detected is consistent with the type of the at least part of the reference text.

4. The method according to claim 1, wherein The additional information includes at least one of the following: a website, a QR code, a barcode, a mobile phone number, and an account name of a predetermined chat software.

5. The method according to any one of claims 1 to 4, wherein: The image features of the image to be detected are obtained based on an image encoder, and the text features of the reference text are obtained based on a text encoder; the image encoder and the text encoder are trained using training samples, the training samples include sample images and sample text groups, and at least two sample texts in the same sample text group describe the same sample image from different dimensions; the image encoder and the text encoder are trained in the following manner: Encoding at least two sample texts in the sample text group based on the text encoder to obtain at least two sample text features; Encoding the sample image based on the image encoder to obtain sample image features; Determining a plurality of loss values ​​according to the similarity between the sample text feature of each sample text in the sample text group and the sample image feature, and the matching relationship between the sample text and the sample image; and The image encoder and the text encoder are trained according to the multiple loss values.

6. The method according to claim 5, wherein: The sample text group includes a first sample text and a second sample text, wherein the first sample text is used to describe the content of the sample image, and the second sample text is used to describe whether the sample image includes additional information and the type of the additional information if the additional information is included.

7. An image detection device comprising: An image feature determination module, used to determine the image features of the image to be detected; a similarity determination module for determining, in response to detecting that both text features of at least one reference text and the image features satisfy a matching condition, a similarity between the image features and the reference features; wherein the reference features are determined based on features of the reference image including the additional information; and a first result determination module, configured to determine a detection result of the image to be detected based on a similarity between the image feature and each reference feature, wherein the detection result indicates whether the image to be detected includes additional information and, if additional information is included, the type of the additional information; The reference text is obtained by combining a type text and a template text, the type text is used to describe the type of additional information, and the template text is used to describe that the image contains additional information; the matching condition includes: the similarity between the text features of the reference text and the image features is not greater than a similarity threshold.

8. The device according to claim 7, wherein The reference characteristics are determined by the following modules: an acquisition module, configured to acquire at least one reference image set, each reference image set including at least one reference image; wherein the reference images in the same reference image set include the same type of additional information, and the types of the additional information in different reference image sets are different from each other; and A fusion module is used to fuse the features of the reference images in each reference image set in the at least one reference image set to obtain the reference features; wherein the reference features correspond one-to-one to the reference image set.

9. The apparatus according to claim 7, further comprising: The second result determination module is used to determine the detection result in response to detecting that the text features of at least part of the reference text in the at least one reference text do not meet the matching condition with the image features, and the detection result indicates that: the image to be detected includes additional information, and the type of the additional information of the image to be detected is consistent with the type of the at least part of the reference text.

10. The device according to claim 7, wherein The additional information includes at least one of the following: a website, a QR code, a barcode, a mobile phone number, and an account name of a predetermined chat software.

11. The device according to any one of claims 7 to 10, wherein: The image features of the image to be detected are obtained based on an image encoder, and the text features of the reference text are obtained based on a text encoder; the image encoder and the text encoder are trained using training samples, wherein the training samples include a sample image and a sample text group, and at least two sample texts in the same sample text group describe the same sample image from different dimensions; the image encoder and the text encoder are trained using the following modules: a first encoding module, configured to encode at least two sample texts in the sample text group based on the text encoder to obtain at least two sample text features; a second encoding module, configured to encode the sample image based on the image encoder to obtain sample image features; a loss determination module, configured to determine a plurality of loss values ​​based on a similarity between a sample text feature of each sample text in the sample text group and a feature of the sample image, and a matching relationship between the sample text and the sample image; as well as A training module is used to train the image encoder and the text encoder according to the multiple loss values.

12. The device according to claim 11, wherein The sample text group includes a first sample text and a second sample text, wherein the first sample text is used to describe the content of the sample image, and the second sample text is used to describe whether the sample image includes additional information and the type of the additional information if the additional information is included.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images