Picture auditing method and device, electronic equipment and storage medium

Abstract

The invention provides a picture auditing method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence. The picture auditing method comprises the following steps: performing character detection and recognition on a picture to be audited by utilizing a character recognition module, and extracting text information in the picture to be audited; the text information is matched with a preset text label, a text matching result is determined, and the text matching result is used for indicating whether the text information is successfully matched with the preset text label or not; the to-be-audited picture is matched with a preset image label, an image matching result is determined, and the image matching result is used for indicating whether the to-be-audited picture is successfully matched with the preset image label or not; and according to the text matching result and the image matching result, determining whether the to-be-audited picture is compliant or not. According to the embodiment of the invention, the accuracy of picture auditing can be effectively improved, and the situation of mistaken killing and missing killing can be reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for image review. Background Art

[0002] With the development of the Internet industry, massive amounts of images are uploaded, downloaded, and disseminated on major Internet platforms, which also brings huge risks to risk control. For example, a large amount of illegal information is hidden in the images, which seriously affects the customer experience and perception. In the past, image review relied heavily on manual review, but in recent years, with the development of artificial intelligence technology, deep learning technology based on massive image samples and large-scale graphics processing unit (GPU) computing has gradually become the mainstream solution in the industry.

[0003] In related technologies, single-mode image data features are mainly used to determine whether an image violates regulations. However, for scenarios with more complex audit standards, such as advertising scenarios, there are differences in audit standards under different circumstances, or the customer's own business has customized requirements, or the audit standard judgment logic is complex. Under complex audit standards, the existing audit and judgment based on single-mode image data features is more likely to result in false positives and false negatives. Summary of the invention

[0004] The present application provides a method, device, electronic device and storage medium for image review, which can effectively improve the accuracy of image review and help reduce the occurrence of false positives and missed positives.

[0005] In a first aspect, an embodiment of the present application provides a method for image review, comprising:

[0006] Using a character recognition module to perform text detection and recognition on the image to be reviewed, and extracting text information from the image to be reviewed;

[0007] Matching the text information with a preset text tag to determine a text matching result, wherein the text matching result is used to indicate whether the text information successfully matches the preset text tag; wherein the preset text tag includes at least one of at least one keyword pre-stored in a text vocabulary, at least one pre-configured regular expression, and at least one text category;

[0008] Matching the image to be reviewed with a preset image tag to determine an image matching result, wherein the image matching result is used to indicate whether the image to be reviewed successfully matches the preset image tag; wherein the preset image tag includes at least one of at least one image sample feature pre-stored in an image library, at least one pre-configured image category, and at least one target image;

[0009] Whether the image to be reviewed is compliant is determined based on the text matching result and the image matching result.

[0010] In a second aspect, an embodiment of the present application provides a device for image review, including:

[0011] A character recognition module is used to detect and recognize characters in the image to be reviewed, and extract text information from the image to be reviewed;

[0012] A text matching unit, used to match the text information with a preset text tag and determine a text matching result, wherein the text matching result is used to indicate whether the text information successfully matches the preset text tag; wherein the preset text tag includes at least one of at least one keyword pre-stored in a text word library, at least one pre-configured regular expression, and at least one text category;

[0013] An image matching unit is used to match the image to be reviewed with a preset image tag and determine an image matching result, wherein the image matching result is used to indicate whether the image to be reviewed successfully matches the preset image tag; wherein the preset image tag includes at least one of at least one image sample feature pre-stored in an image library, at least one pre-configured image category, and at least one target image;

[0014] A determination unit is used to determine whether the image to be reviewed is compliant based on the text matching result and the image matching result.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the method of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the method of the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which enable a computer to execute the method of the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program, which enables a computer to execute the method of the first aspect.

[0019] Through the above-mentioned technical scheme, the embodiment of the present application obtains image matching results by matching the image to be reviewed with preset image tags, and obtains text matching results by matching the text information in the image to be reviewed with preset text tags, and then comprehensively analyzes the image matching results and text matching results of the image to be reviewed to determine whether the image is in violation of regulations. Compared with the prior art review method based only on single-modal information of text or images, the embodiment of the present application can effectively improve the accuracy of image review, which is conducive to reducing the misjudgment of compliant images and the omission of illegal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A A schematic diagram of the image to be reviewed;

[0021] Figure 1B Another schematic diagram of the image to be reviewed;

[0022] Figure 1C Another schematic diagram of the image to be reviewed;

[0023] Figure 1D Another schematic diagram of the image to be reviewed;

[0024] Figure 1E Another schematic diagram of the image to be reviewed;

[0025] Figure 2 A schematic diagram of an application scenario of the solution of an embodiment of the present application;

[0026] Figure 3 is a schematic flow chart of a method for image review according to an embodiment of the present application;

[0027] Figure 4A A schematic diagram of a text matching strategy according to an embodiment of the present application;

[0028] Figure 4B is a schematic flow chart of another method for image review according to an embodiment of the present application;

[0029] Figure 5 A schematic diagram of a model training process according to an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the network structure of the TextCNN model;

[0031] Fig. 7A is another schematic diagram of a model training process according to an embodiment of the present application;

[0032] Figure 7B is a schematic flow chart of another method for image review according to an embodiment of the present application;

[0033] Figure 8 A schematic diagram of the network structure of the MobileNetV3 model;

[0034] Fig. 9 A schematic diagram of the network structure of the YOLOX model;

[0035] Fig.10 A schematic diagram of a configuration interface provided in an embodiment of the present application;

[0036] Fig.11 A schematic diagram of a network architecture for image review provided in an embodiment of the present application;

[0037] Fig.12 Another schematic diagram of the image to be reviewed;

[0038] Fig.13 is a schematic flow chart of another method for image review according to an embodiment of the present application;

[0039] Fig.14 Another schematic diagram of the network architecture for image review provided in an embodiment of the present application;

[0040] Fig.15 A schematic diagram of the structure of the Transformer model;

[0041] Fig.16 A schematic diagram of a network architecture provided in an embodiment of the present application;

[0042] Fig.17 is a schematic block diagram of a device for image review according to an embodiment of the present application;

[0043] Fig.18 is a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0045] It should be understood that in the embodiment of the present application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based only on A, and B can also be determined based on A and / or other information.

[0046] In the description of the present application, unless otherwise specified, "at least one" means one or more, and "plurality" means two or more than two. In addition, "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0047] It should also be understood that the first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects, without any distinction of order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0048] It should also be understood that the specific features, structures or characteristics related to the embodiments in the specification are included in at least one embodiment of the present application. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0049] In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or device.

[0050] The embodiments of the present application are applied to the field of artificial intelligence technology.

[0051] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0052] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0053] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0054] The embodiments of the present application may involve natural language processing (NLP) in artificial intelligence technology, which is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes technologies such as text processing, semantic understanding, machine translation, robot question answering, and knowledge graphs.

[0055] The embodiments of the present application may involve computer vision (CV) technology in artificial intelligence technology. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify, monitor and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0056] The embodiments of the present application may also involve machine learning (ML) in artificial intelligence technology. ML is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0057] The present application embodiment may also relate to multimodal learning (Multi-modalLearning) in artificial intelligence technology, which refers to the use of multiple different types of data as input in machine learning to improve the accuracy and performance of the model. These data can be information of different types and structures from different sources, such as text, images, videos, audio, etc. In multimodal learning, different types of data need to be fused and integrated to extract useful feature information and reduce redundant information. By utilizing multiple different types of data, more comprehensive and accurate information can be obtained, thereby improving the performance and robustness of the model.

[0058] At present, single-mode text or image data features are mainly used to determine whether an image is in violation of regulations. However, for scenarios with more complex audit standards, such as advertising scenarios, when auditing and judging based on single-mode image data features under complex audit standards, the situation of false positives and false negatives is more serious. For example, the complexity of audit standards can be reflected in the following aspects.

[0059] 1) There are differences in the audit standards in different situations, such as Figure 1A It is illegal for an image of a specific uniform to appear in an e-commerce advertisement, but the image of the specific uniform is not allowed in Figure 1B If a single-mode image recognition model is used to identify a picture of a specific uniform and then review it as a violation, it is easy to mistakenly kill pictures of TV drama advertisements.

[0060] 2) Customers have many customized needs for their own business. For example, some mobile phone manufacturers (first-brand mobile phones) do not want the advertising materials of competing mobile phones (such as second-brand mobile phones) to appear on their own platforms, but at the same time they do not want the advertisements of their own mobile phones to be blocked. Figure 1C The first brand mobile phone advertising pictures are not illegal advertising pictures. Figure 1DThe second brand mobile phone advertising picture is a violation of the regulations. At this time, if a single-mode image recognition model is used to identify the mobile phone picture and review it as a violation, it is easy to mistakenly kill the mobile phone advertisement of the company.

[0061] 3) The logic of the review criteria is complex. For example, for red envelope transfer ads, the client's criteria are:

[0062] a. The specific red envelope amount does not appear in the picture, which is a normal picture;

[0063] b. If the specific red envelope amount appears in the picture and a red envelope reminder is added, it is a normal picture;

[0064] c. If the specific red envelope amount appears in the picture and no red envelope reminder is added, it is an illegal picture;

[0065] d. The red envelope amount in the picture exceeds 100 yuan, which is an illegal picture.

[0066] In addition to combining the information of the image and text to participate in the violation analysis of the image and determine whether "red envelope image + red envelope prompt" exist at the same time, this type of user standard also needs to provide fast and configurable rule logic to implement logical judgment of rules such as "amount exceeds 100". Figure 1E A schematic diagram showing an image of a red envelope promotion advertisement that does not violate regulations, which includes the red envelope prompt "Up to 100 yuan, the specific amount is subject to actual collection", and the amount is less than 100 yuan.

[0067] In order to solve the above technical problems, the embodiments of the present application provide a method, device and storage medium for image review, which can effectively improve the accuracy of image review and help reduce the misjudgment of compliant images and the omission of illegal images.

[0068] Specifically, the character recognition module can be used to detect and recognize text on the image to be reviewed to extract text information from the image to be reviewed; match the text information with a preset text label to determine a text matching result, which is used to indicate whether the text information successfully matches the preset text label; match the image to be reviewed with a preset image label to determine an image matching result, which is used to indicate whether the image to be reviewed successfully matches the preset image label; determine whether the image to be reviewed is compliant based on the text matching result and the image matching result.

[0069] The embodiment of the present application obtains an image matching result by matching the image to be reviewed with a preset image tag, and obtains a text matching result by matching the text information in the image to be reviewed with a preset text tag, and then comprehensively determines whether the image to be reviewed is in violation of regulations by combining the image matching result and the text matching result. Compared with the prior art review method based only on single-modal information of text or images, the embodiment of the present application can effectively improve the accuracy of image review, which is conducive to reducing the misjudgment of compliant images and the omission of illegal images.

[0070] The embodiment of the present application combines the image matching results and text matching results of the image to be reviewed, and can be applied to complex image review standard scenarios, such as advertising image review scenarios, where there are differences in review standards under different circumstances, or the customer's own business has customized requirements, or the review standard judgment logic is complex. Figure 1A or Figure 1B For a specific uniform image, it can be judged as normal when the specific uniform image is identified and the text information is a TV drama advertisement type label, and it can be judged as illegal when the specific uniform image is identified and the text information is an e-commerce advertisement type label. Figure 1C or Figure 1D For example, when a mobile phone advertisement image is identified and the text information is a first brand type label, it can be judged as normal, and when a mobile phone image is identified and the text information is a second brand type label, it can be judged as illegal. Figure 1E In the red envelope advertisement picture, it can be judged as normal if the red envelope picture is identified and the text information contains "subject to actual collection" and the red envelope amount is less than 100 yuan.

[0071] The embodiments of the present application can be applied to products (such as applications) for image review, which can provide services to customers through the calling method of application programming interface (API) or software development kit (SDK). The embodiments of the present application can support advertising, social platforms, online education, e-commerce platforms, game applications and other scenarios to provide users with the ability to review image content, helping users to effectively identify images containing illegal information.

[0072] Figure 2 A schematic diagram showing an application scenario of an embodiment of the present application.

[0073] like Figure 2 As shown, the application scenario involves a server 1 and a terminal device 2, and the terminal device 2 can communicate data with the server 1 through a communication network. The server 1 can be a background server of the terminal device 2.

[0074] Exemplarily, the terminal device 2 may refer to a type of device that has a rich human-computer interaction mode, has the ability to access the Internet, is usually equipped with various operating systems, and has strong processing capabilities. The terminal device may be a smart phone, a tablet computer, a portable laptop, a desktop computer, a wearable device, a vehicle-mounted device, and other terminal devices, but is not limited thereto. Optionally, in the embodiment of the present application, the terminal device 2 is installed with an image review application or an application with an image review function.

[0075] Server 1 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also become a node of the blockchain.

[0076] The server may be one or more servers. When there are multiple servers, there are at least two servers for providing different services, and / or there are at least two servers for providing the same service, such as providing the same service in a load balancing manner, which is not limited in the embodiments of the present application.

[0077] The terminal device and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this. This application does not limit the number of servers or terminal devices. The solution provided by this application can be completed independently by the terminal device, or by the server, or by the terminal device and the server in cooperation, and this application does not limit this.

[0078] It should be understood that Figure 2 This is only an exemplary description and does not specifically limit the application scenarios of the embodiments of the present application. Figure 2 One terminal device and one server are shown as an example, but other numbers of terminal devices and servers may actually be included, and this application does not impose any limitation on this.

[0079] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0080] Figure 3FIG. 3 is a schematic flow chart of a method 300 for image review according to an embodiment of the present application. The method 300 can be executed by any electronic device with data processing capability. For example, the electronic device can be implemented as a server or a terminal device, such as the server 1 or the terminal device 2 in FIG. 1 . The present application does not limit this. Figure 3 As shown, method 300 includes steps 310 to 340 .

[0081] 310, using a character recognition module to perform text detection and recognition on the image to be reviewed, and extracting text information from the image to be reviewed.

[0082] Exemplarily, the images to be reviewed can be images uploaded by users on the platform in scenarios such as advertising, social platforms, online education, e-commerce platforms, and game applications, or images in videos uploaded by users on the platform, and this application does not limit this. Optionally, the images to be reviewed can be obtained through an API interface or SDK. The images to be reviewed can include multimodal information such as images and text.

[0083] In some embodiments, the character recognition module can be specifically implemented as an optical character recognition (OCR) system, which can obtain text information appearing in the image to be reviewed by performing text detection and text recognition on the image to be reviewed. Exemplarily, the text information can include text objects, such as Chinese characters, foreign characters, alphabetic characters, numeric characters, and identifiers with specific semantics, etc., without limitation.

[0084] 320, matching the text information with the preset text tag, and determining a text matching result, wherein the text matching result is used to indicate whether the text information successfully matches the preset text tag. The preset text tag may include at least one of at least one keyword pre-stored in a text word library, at least one pre-configured regular expression, and at least one text category.

[0085] When the text information matches the preset text tag, the text matching result is used to indicate that the text information successfully matches (i.e. hits) the preset text tag; when the text information does not match the preset text tag, the text matching result is used to indicate that the text information does not successfully match (i.e. misses) the preset text tag.

[0086] Exemplarily, the preset text label that matches the text information can be determined based on the text matching strategy. The text matching strategy may include a text vocabulary strategy, a text rule strategy, and a deep learning strategy, etc. The text vocabulary strategy indicates that a text label that matches the text information is retrieved from the text vocabulary, and the keywords in the text vocabulary can be used as preset text labels. The text rule strategy indicates that a text label that matches the text information is obtained by matching according to at least one pre-configured regular expression, and the text content corresponding to the regular expression can be used as the preset text label. The deep learning strategy indicates that the text information is classified by a text classification model to obtain a text label that matches the text information, and at least one text category corresponding to the text classification model can be used as a preset text label. The text matching result is the text content review result. Figure 4A As shown, based on the text information in the image to be reviewed extracted in step 310, the text vocabulary strategy, text rule strategy and deep learning strategy can be flexibly used to obtain the text label corresponding to the text information as the text matching result.

[0087] In some embodiments, the preset text label includes a text label for indicating that the target type information in the image is compliant. The target type information in the image may refer to a preset specific type of information, which is compliant in some scenarios and not compliant in some scenarios. The target type information may be set by the customer to meet the customer's business needs. For example, Figure 1A and Figure 1B The specific uniforms in Figure 1C and Figure 1B If a single-modal image data feature is used, it is impossible to determine whether the target type information is in violation of the rules.

[0088] In some embodiments, the text word library includes at least one keyword for indicating compliance of target type information in the image. Figure 4B , in step 321, whether there is a keyword matching the content of the text information in at least one keyword used to indicate compliance of the target type information in the image, and obtaining a text matching result. When the text information matches a certain keyword, the text information hits the keyword.

[0089] As an implementable manner, the text word library may include a custom word library, which includes at least one user-defined keyword for indicating compliance of target type information in the image. The custom word library can be configured separately for different customers, making it convenient for customers to add customized word libraries.

[0090] Exemplarily, the keywords used to indicate compliance of target type information can be keywords configured by the user to indicate compliance of the target type information in the picture. For example, a video platform customer can configure its "video platform name" as a keyword in the custom word library to indicate that the specific uniform in the picture is a TV drama advertisement picture, that is, a compliant picture; for another example, a mobile phone manufacturer customer can configure its "mobile phone brand name" as a keyword in the custom word library to indicate that the mobile phone in the picture is a self-branded advertisement picture, that is, a compliant picture. For another example, a red envelope promotion customer can configure the red envelope reminder "subject to actual collection" as a keyword in the custom word library to indicate that the red envelope picture in the picture is a compliant advertisement picture.

[0091] Therefore, the embodiment of the present application can adapt to the customer's complex standard image content review scenarios and meet the customer's customized business needs by configuring keywords in the text vocabulary to indicate the compliance of the target type information in the image.

[0092] Optionally, the above-mentioned text word library may also include at least one keyword for indicating violation information. As an implementable manner, the text word library may also include a public word library, which includes at least one keyword for indicating violation information. Specifically, the public word library may be a universal word library for various types of violation texts, which may be effective for the image review policies of all customers and may be used to solve the problem of universality. Exemplarily, the violation keywords in the public word library may be keywords containing violation information. When the text information matches such keywords, that is, when the text information corresponds to violation information, it may be directly determined that the image is in violation.

[0093] Optionally, before searching in the text vocabulary, the text information may be preprocessed, including but not limited to escaping emoticons, converting English letters to lowercase, converting traditional Chinese to simplified Chinese, etc., so that the matching result of the text information in the text vocabulary is more accurate.

[0094] Optionally, when configuring keywords, you can also use common text, pinyin, English, combination words, exclusion words and other types of keyword configurations to meet the needs of various business scenarios. For example, when the keyword is configured as English, the matching words in the specified text information must be English to take effect. When the keyword is configured as a combination word, it will take effect only when the specified text contains multiple combination words at the same time.

[0095] In some embodiments, the at least one regular expression includes at least one regular expression for indicating that the target type information in the image is compliant. Figure 4B, a text matching result can be obtained by matching the text information with at least one regular expression used to indicate compliance of the target type in the image in step 322. When the text information matches a regular expression, the text information hits the regular expression.

[0096] Specifically, in scenarios with complex audit standards, there are still some situations that cannot be completely solved by simply using a text vocabulary, such as the Figure 1E In order to facilitate the recall of pictures with more complex customer customization requirements, the embodiment of the present application can perform text recall based on the text rule strategy (such as regular matching strategy).

[0097] Exemplarily, the regular expression used to indicate compliance of the target type information may be a string configured by the user to indicate compliance of the target type information in the image. For example, in addition to configuring the red envelope prompt "subject to actual collection" as a keyword in the custom word library, a red envelope promotion customer may also configure "amount less than 100 yuan" as a regular expression. When the red envelope prompt and the regular expression are matched at the same time (for example, when the red envelope amount is 65.72), the red envelope in the image is a compliant image.

[0098] Therefore, the embodiment of the present application can adapt to customer standard complex image content review scenarios and meet customer customized business needs by configuring a regular expression for indicating compliance of target type information in the image.

[0099] Optionally, before text rule matching, the text information may be preprocessed, including but not limited to escaping emoticons, converting English letters to lowercase, converting traditional Chinese to simplified Chinese, etc., so that the text rule matching result of the text information is more accurate.

[0100] In some embodiments, when a deep learning strategy is used, see Figure 4B , the text information can be input into the text classification model through step 323 to determine the text category that matches the text information in the at least one text category as the text matching result. The at least one text category includes a label type for indicating that the target type information of the image is compliant, a label type for indicating that the text information is illegal, and a label type for indicating that the text information is compliant. The text classification model is trained based on the text sample and the annotation information of the text sample.

[0101] For example, the text classification model can be a lightweight text convolutional neural network (TextCNN) model, which can facilitate the real-time deployment of image content review capabilities online and reduce the time consumption of image analysis. The data source for model training is the manually annotated text samples submitted by customers for inspection, and the annotation information can be the type label of the text submitted for inspection.

[0102] Figure 5 A schematic diagram of the training process of the model is shown. As shown in the figure, the text to be reviewed 510 is processed by the text preprocessing module 520, such as normalization of special characters, regular rule matching, and removal of invalid symbols. After data enhancement, it is further vectorized by the text vectorization module 530, such as feature extraction and embedding vectorization of the text to obtain the embedding vector of the text. Then, the embedding vector of the text is input into the TextCNN model 540 to obtain a multi-classification label score 550. In some embodiments, the review result of the text in the picture can be obtained based on the label score. For example, when the label score of a certain label is 0.9, it can be determined that the text category matched by the text information is this type of label.

[0103] Figure 6 A schematic diagram of the network structure of the TextCNN model is shown. As shown in the figure, the embedding layer extracts the character features, pinyin features, and glyph features of the text to be reviewed into a deep feature vector; the convolutional layer performs a one-dimensional convolution operation on the deep feature vector to select features and reduce the number of parameters; the pooling layer performs a pooling operation on the result after convolution, so that the network has a larger receptive field and avoids overfitting; the fully connected layer outputs the label score of each review category.

[0104] 330, matching the image to be reviewed with a preset image tag, and determining an image matching result, wherein the image matching result is used to indicate whether the image to be reviewed successfully matches the preset image tag; wherein the preset image tag includes at least one of at least one image sample feature pre-stored in the image library, at least one pre-configured image category, and at least one target image.

[0105] When the image to be reviewed matches the preset image tag, the image matching result is used to indicate that the image to be reviewed successfully matches (i.e. hits) the preset image tag; when the text information does not match the preset image tag, the image matching result is used to indicate that the image to be reviewed does not successfully match (i.e. misses) the preset image tag.

[0106] Exemplarily, the preset image label that matches the image to be reviewed can be determined according to the image matching strategy. The image matching strategy can include a deep learning strategy, which indicates that an image label that matches the image to be reviewed is obtained using an image deep learning model (such as an image similarity model, an image classification model, an image detection model, etc.). The image matching result is the image content review result.

[0107] Fig. 7AA schematic diagram of the training process of the model is shown. As shown in the figure, the image to be reviewed 610 is processed by the image data enhancement module 620, such as image rotation and flipping, image cropping, image deformation, image noise generation, image blur, image color change, etc., and then pre-processed by the image pre-processing module 630 such as image size normalization, image grayscale, etc., and then input into the deep learning model 640 to obtain the image review result 650. Exemplarily, the image review result may include the image label of the image to be reviewed. Exemplarily, the deep learning model may include an image similarity model, an image classification model, or an image detection model, etc., without limitation.

[0108] In some embodiments, the preset image tag may include an image tag for indicating that the image to be reviewed includes target type information. Specifically, the target type information can be found in the description above and will not be repeated here. When the image to be reviewed successfully matches the preset image tag of this type, it is necessary to further determine whether the image to be reviewed is compliant based on the preset type tag corresponding to the text information.

[0109] In some embodiments, see Figure 7B , in step 331, the image to be reviewed can be input into the image similarity model, and the similarity matching between the image to be reviewed and at least one image sample feature in the image library can be performed to obtain an image matching result. The at least one image sample feature includes an image feature of an image containing target type information and an image feature of an image containing violation information.

[0110] Specifically, for the image similarity model, according to the actual audit scenario, the target image can be selected as the image sample, and the feature vector of the sample image can be extracted using the deep learning model as the target image library. Then, the same model is used to extract the feature vector of the input image to be audited, and the similarity is matched with the features in the target image library. If it is greater than a certain threshold, it is determined to be the target image. Exemplarily, the target image may include illegal images, in which case the image sample features include the image features of the image containing illegal information. The target image may also include the image containing the target type information as described above, in which case the image sample features include the image features of the image containing the target type information. When the target image includes illegal images, the target image library is a black image library, and at this time, images whose feature similarity with the black image library is greater than a certain threshold can be processed as illegal images.

[0111] In some embodiments, see Figure 7B, the image to be reviewed can be input into the image classification model through step 332 to determine the image type that matches the image to be reviewed in the at least one image category as the image matching result. The at least one image category includes a label type for indicating that the image contains target type information, a label type for indicating that the image contains illegal information, and a label type for indicating that the image is compliant.

[0112] For example, the image classification model can use a lightweight mobile network model (such as MobileNetV3), which can facilitate the online real-time deployment of image content review capabilities and reduce the time consumption of image analysis. The data source for model training is the manually annotated customer-submitted image samples, and the annotation information can be the type label of the submitted image.

[0113] Figure 8 A schematic diagram of the network structure of the MobileNetV3 model is shown. As shown in the figure, the image feature 710 (e.g., the size of 3*224*224) of the image to be reviewed is input and processed in sequence by the convolution layer 720, the separable convolution layer block (bneck) layer 730, the pooling layer 740 and the fully connected layer 750, and the label score 760 is output. Among them, the bneck layer 730 is the core module of the MobileNetV3 model, which can realize the channel separable convolution + channel attention mechanism and residual connection.

[0114] In some embodiments, see Figure 7B , in step 333, the image to be reviewed may be input into the image detection model, and the image to be reviewed may be matched with at least one target image to obtain an image matching result. The at least one target image may include an image containing the target type information and an image containing violation information.

[0115] Specifically, in the image content review scenario, the illegal content may only account for a small part of the entire image material. At this time, the effect of simply using the image classification model is not ideal. Therefore, the embodiment of the present application introduces an image detection model to further improve the overall effect. For example, the image detection model can select the YOLOX model, such as Fig. 9As shown in the figure, it includes a prediction branch decoupling head, which first reduces the dimension of the incoming feature map to 256 through a convolutional layer (Conv1×1), and then sends it to two branches, one branch is responsible for predicting the bounding box (bbox) category (i.e. Cls. in the figure, the size can be H×W×C), and the other branch is responsible for predicting the bbox position (i.e. Reg. in the figure, the size can be H×W×4) and overlap (i.e. IoU. in the figure, the size can be H×W×1). Using this decoupling head can greatly shorten the convergence speed on the one hand, and on the other hand, it brings a 4.2% average precision (AP) improvement compared to the end-to-end method without a decoupling head.

[0116] In some embodiments, the image similarity threshold, text classification threshold, and image classification threshold can all be customized to meet the customized needs of customers. Fig.10 A schematic diagram of the configuration interface is shown, in which you can choose to turn on violation type 1, select the various violation subcategories that need to be identified, and configure the violation score threshold for each violation subcategory. Among them, the audit violation state BLOCK (i.e. interception) corresponds to severe violations, and customers should intercept. The score threshold of the BLOCK state is set at 70-100 in the figure, that is, severe violations with a violation score threshold of 70-100 are intercepted. Optionally, the embodiment of the present application can also set the audit violation state to REVIEW (i.e. suspected), and the audit violation state REVIEW corresponds to minor violations, and customers can decide whether to intercept the process. In addition, the violation score threshold is 0-40 and is not a violation.

[0117] 340, based on the text matching results and the image matching results, determine whether the image to be reviewed is compliant.

[0118] Specifically, in this step, the text matching results in step 320 and the image matching results in step 330 can be integrated, and the audit results of the text and the image can be flexibly combined to determine whether the image to be audited is compliant. Based on the text matching results and the image matching results, it is possible to improve the accuracy of the audited image to a certain extent when the customer's audit standards are complex. Exemplarily, the audit results of the image can include normal, suspected, and illegal, etc., without limitation.

[0119] In some embodiments, when the preset text tag includes a text tag for indicating that the target type information is compliant, and the preset image tag includes an image tag for indicating that the picture to be reviewed includes the target type information, if the text matching result is used to indicate that the text information successfully matches the preset text tag, and the image matching result is used to indicate that the picture to be reviewed successfully matches the preset image tag, then the picture to be reviewed is determined to be compliant.

[0120] That is, when the image to be reviewed contains an image of target type information, and the text information in the image to be reviewed indicates that the target type information is compliant, the image to be reviewed can be judged as a normal (ie compliant) image. Table 1 shows several specific examples of image review results.

[0121]

[0122] Specifically, for Figure 1B For the pictures to be reviewed in the video, the image label is "specific uniform". The picture contains target type information (i.e., specific uniform). At the same time, the text information hits the keyword "video platform name" in the text vocabulary, indicating that the specific uniform in the picture is a scene from a TV series. Therefore, the review result of the picture is normal, that is, the specific uniform in the picture is the specific uniform that appears in the TV series advertisement.

[0123] for Figure 1C For the image to be reviewed in the image, the image label is "mobile phone", the image contains the target type information (i.e. mobile phone), and the text information hits the keyword "first mobile phone brand name" in the text vocabulary, indicating that the mobile phone in the image is a self-owned brand. Therefore, the review result of the image is normal, that is, the mobile phone in the image is the customer's own mobile phone brand.

[0124] for Figure 1E For the image to be reviewed in , the image label is "red envelope", which satisfies the image containing the target type information (i.e., red envelope). At the same time, the text information hits the keyword "subject to actual receipt" in the text vocabulary, and hits the regular rule "red envelope amount 65.72", indicating that the red envelope meets the review standards. Therefore, the review result of the image is normal, that is, the red envelope in the image has a prompt and the amount is less than 100 yuan.

[0125] In some embodiments, when the preset text tag includes a text tag for indicating that the target type information is compliant, and the preset image tag includes an image tag for indicating that the image to be reviewed includes the target type information, if the text matching result is used to indicate that the text information does not successfully match the preset text tag, and the image matching result is used to indicate that the image to be reviewed successfully matches the preset image tag, then the image to be reviewed is determined to be non-compliant. In other words, when the image to be reviewed satisfies the image containing the target type information, and the text information in the image to be reviewed does not contain text indicating that the target type information is compliant, the image to be reviewed can be judged as an illegal image.

[0126] For example, for Figure 1AFor the image to be reviewed in , the image label can be "specific uniform", but the text information contains the e-commerce platform information, indicating that the specific uniform in the image is a picture in the e-commerce advertisement, and it cannot indicate that the specific uniform is a compliant image. Therefore, the review result of the image is illegal. For another example, for Figure 1D For the image to be reviewed, the image label is "mobile phone", but the text information contains information related to the second mobile phone brand, which means that the mobile phone in the picture is not its own brand and cannot indicate that the mobile phone is a compliant picture. Therefore, the review result of the picture is violation.

[0127] In some embodiments, when the preset text tag includes a tag type for indicating that the text information is in violation of regulations, if the text matching result is used to indicate that the text information successfully matches the preset text tag, it is determined that the image to be reviewed is non-compliant.

[0128] In some embodiments, when the preset image label includes a label type for indicating that the image contains illegal information, if the image matching result is used to indicate that the image to be reviewed successfully matches the preset image label image, the image to be reviewed is determined to be non-compliant.

[0129] That is, when at least one of the image or text in the image to be reviewed contains illegal information, the image to be reviewed can be judged as an illegal image. Exemplarily, the illegal information can include general illegal information, such as various prohibited contents or uncomfortable images or text information.

[0130] Fig.11 A schematic diagram of the network architecture of image audit is shown. As shown in the figure, the image 810 to be audited can be input into the image deep learning model 820 to obtain the image content audit result, and the image content audit result can include the image label matched by the image output by the deep learning model. At the same time, the image 810 can be processed by OCR to obtain the text information 830 therein, and the text information is obtained according to the text matching strategy module 840, such as through text vocabulary matching, text rule matching and text deep learning model, The text content audit result may include at least one of the text label matched for this vocabulary, the text label matched by the text rule, and the text label of the text deep learning model. Then, the image content audit result and the text content audit result are fused by the image-text fusion module 850 to obtain the image-text fusion result, and the audit result 860 can be obtained according to the image-text fusion result. The audit result 860 may include normal, suspected and illegal, etc., without limitation.

[0131] Among them, the text content review results may include hitting preset text tags, such as hitting keywords used to indicate compliance of target type information in the image, keywords used to indicate illegal information, regular expressions used to indicate compliance of target type information in the image, tag types used to indicate compliance of text information in the image, and tag types used to indicate illegal information in the text, etc., without limitation. The image content review results include hitting preset image tags, such as hitting image features of images containing target type information, image features of images containing illegal information, tag types used to indicate images containing the target type information, tag types used to indicate images containing illegal information, target images containing the target type information, and target images containing illegal information, etc., without limitation.

[0132] In some embodiments, the fusion results include four types: hitting a preset text tag alone, hitting a preset image tag alone, hitting both preset text and image tags, and hitting both unnamed preset text and image tags.

[0133] In the case of hitting a preset text tag alone, when the text information hits a keyword indicating compliance of the target type information in the image, a regular expression indicating compliance of the target type information in the image, or a tag type indicating compliance of the target type information in the image, the image to be reviewed can be judged as a normal image. When the text information hits a keyword indicating violation information or a tag type indicating violation of the text information, the image to be reviewed can be judged as a violation image.

[0134] In the case of hitting a preset image tag alone, when the image feature of a picture containing target type information, the tag type used to indicate that the picture contains the target type information, or the target image containing the target type information is hit, the picture to be reviewed can be judged as an illegal picture because the preset text tag is not hit. When the image feature of a picture containing illegal information, the tag type used to indicate that the picture contains illegal information, or the target image containing illegal information is hit, the picture to be reviewed can be directly judged as an illegal picture.

[0135] For the case where preset text and image tags are hit at the same time, when the image hits the image features of the image containing the target type information, the tag type used to indicate that the image contains the target type information, or the target image containing the target type information, and the text information hits the keywords used to indicate that the target type information in the image is compliant, the regular expression used to indicate that the target type information in the image is compliant, or the tag type of the text used to indicate that the target type information in the image is compliant, the image to be reviewed can be judged as a normal image. For other cases where the text image model is hit at the same time, as long as the image tag that satisfies the image matching corresponds to the violation information, or the text tag that matches the text information corresponds to the violation information, the image to be reviewed is a violation image.

[0136] In the case where both the preset text and image tags are not hit, since no target type information or violation information is involved, the image to be reviewed can be judged as a normal image.

[0137] Therefore, the embodiment of the present application obtains image matching results by matching the image to be reviewed with preset image tags, and obtains text matching results by matching the text information in the image to be reviewed with preset text tags, and then comprehensively analyzes the image matching results and text matching results of the image to be reviewed to determine whether the image is in violation of regulations. Compared with the prior art review method based only on single-modal information of text or images, the embodiment of the present application can effectively improve the accuracy of image review, which is conducive to reducing the misjudgment of compliant images and the omission of illegal images.

[0138] Furthermore, the embodiments of the present application provide text thesaurus, text rules and other strategies, and support text and image combination strategy configuration, allowing for quick and flexible configuration of thesaurus and rules to support new standards based on specific customer scenarios without relying on updates and iterations of deep learning models, which can help reduce the response time to customer review needs.

[0139] In some embodiments, when the probability of the image to be reviewed being in violation is determined to be in a specific interval based on the text matching results and the image matching results, the image to be reviewed can be judged as a suspected image. For example, when the probability that the text information is a label type used to indicate that the text information is in violation is in a preset specific interval based on the text classification model, or the feature similarity between the image to be reviewed and the illegal image sample is determined to be in a specific interval based on the image similarity model, or the probability that the image is a label type containing illegal information is determined to be in a preset specific interval based on the image classification model, or the probability that the image to be reviewed contains the target image is determined to be in a specific interval based on the image detection model, it can be determined that the probability of the image to be reviewed being in violation is in a specific interval, and the image to be reviewed is a suspected image at this time. Suspected images may be normal images or illegal images.

[0140] For example, some violation categories may be difficult to review. In this case, it is difficult to determine whether an image is in violation based only on a single-mode text image or in combination with the text and image content review results. Fig.12 In the case of a specific type of scale violation in the example, due to the characteristics of this specific type of edge-wiping, the scales of the specific type of edge-wiping pictures and specific type of edge-wiping text in the picture are sensitive to a certain extent but have not reached the violation threshold. At this time, neither the single-mode review of the existing solution nor the results of the direct combination of picture and text review can solve this problem well.

[0141] In some embodiments, see Fig.13 , the embodiment of the present application can also use multimodal graphic features to determine whether the image is compliant through the following steps 350 to 380. It should be understood that steps 350 to 380 can be performed after step 340 to further determine whether the suspected image is compliant. Steps 350 to 380 can also be performed separately to determine whether the image to be reviewed is compliant, and this application does not limit this.

[0142] 350, extract features from the text information to obtain text features of the text information.

[0143] In some embodiments, the text features of the text information can be extracted by a TextCNN model. Specifically, the TextCNN model can refer to the relevant description above, which will not be repeated here.

[0144] See also Fig.14 , the image to be reviewed 910 can be subjected to OCR processing to obtain text information 930 in the image, and the text information 930 can be input into the TextCNN model 940 to extract text features. Among them, the image to be reviewed can include a suspected image, without limitation. Specifically, the text information is similar to that in step 310, and the relevant description can be referred to above.

[0145] 360, extract features from the images to be reviewed and obtain image features of the images to be reviewed.

[0146] In some embodiments, the image features of the image to be reviewed can be extracted through the MobileNetV3 model. Specifically, the MobileNetV3 model can refer to the relevant description above, which will not be repeated here.

[0147] Continue to see Fig.14 , the image to be reviewed 910 can be input into the MobileNetV3 model 920 to extract the image features of the image to be reviewed.

[0148] 370, input the text features and the image features into the neural network model, and fuse the text features and the image features to obtain multimodal fusion features.

[0149] Continue to see Fig.14 , the text features and image features can be input into the multimodal feature fusion model 950, and the text features and image features are fused to obtain multimodal fusion features.

[0150] Optionally, before inputting the text features and the image features into the neural network model, the text features and the image features may be concatenated, and the concatenated features may be input into the neural network model for feature fusion to obtain multimodal fusion features.

[0151] In some embodiments, the neural network model includes a Transformer model. Fig.15 A schematic diagram showing the structure of the Transformer model. As shown in the figure, the Transformer model includes multiple (such as N) stacked multi-head attention modules, residual and normalization (Add&Norm) layers, and forward feedback (FeedForward) layers, which gradually fuse the original image features and text features from coarse to fine granularity, and finally obtain image and text fusion features, that is, multimodal fusion features. The multi-head attention module provides multiple feature representation subspaces, so that the Transformer model can fuse image features and text features at different levels, making the fused image and text features more accurate.

[0152] 380, input the multimodal fusion features into the classifier to obtain a classification label of the image to be reviewed, and the classification label is used to indicate whether the image to be reviewed is compliant.

[0153] For example, the multimodal fusion features can be input into the Softmax layer to obtain the classification label of the image to be reviewed. Since the fused multimodal fusion features can be more accurate, the classification label can more accurately indicate whether the image to be reviewed is compliant.

[0154] In the actual training process of the model, the TextCNN model, MobileNetV3 model and Transformer model can be trained in a phased training manner. For example, the TextCNN model can be first trained based on text samples and annotations, and the MobileNetV3 model can be trained based on image samples and annotations. After the two models are trained, the Transformer model can be trained. Compared with end-to-end training, this training method has the following advantages: a) The number of samples required is small. For example, only a small number of data samples are needed in the training stage of the Transformer model, and a large amount of manpower is not required to annotate samples; b) In some embodiments, the TextCNN model and the MobileNetV3 model are model tasks that need to be trained separately. In the previous text content review and image content review process, it is equivalent to having completed their training. Therefore, only the Transformer model needs to be trained here, and the training takes less time and the iteration is faster.

[0155] Therefore, the embodiment of the present application deeply fuses the image features of the image to be reviewed and the text features of the text information in the image to be reviewed to obtain multimodal fusion features that can more accurately and effectively perceive the illegal features in the image and make a judgment on whether it is illegal. The embodiment of the present application can effectively improve the accuracy of image review for images with borderline characteristics, and is suitable for the review of images of some illegal categories that are more difficult to review.

[0156] Fig.16 A schematic diagram of a network architecture provided by an embodiment of the present application is shown. As shown in the figure, the network architecture includes a model strategy center, a model strategy configuration, a content review category, and a customer scenario from bottom to top. Among them, in the model strategy center, a text matching strategy and an image matching strategy are provided, wherein the text matching strategy further includes strategies such as a text vocabulary strategy, a text rule strategy, and a text classification model strategy, and the image matching strategy further includes models such as an image classification model, an image detection model, and an image similarity model, so as to adapt to customer needs in different content review categories and customer scenarios.

[0157] In the model strategy configuration, you can configure single-modal strategy and multi-modal strategy. Among them, the single-modal strategy can use text matching strategy or image matching strategy to review the image content. Among them, the text matching strategy is at least one of the text vocabulary strategy, text rule strategy and text model strategy, and the image matching strategy is at least one of the image similarity model strategy, image classification model strategy and image detection model strategy. In this model strategy configuration, it can also support customized configuration of threshold parameters such as image similarity threshold, text classification threshold, image classification threshold, etc. for different customers, and it can also support the selection of violation subtypes and violation score thresholds suitable for the current customer scenario.

[0158] The multimodal strategy includes a text-image combination strategy and a multimodal model strategy. Among them, the text-image combination strategy directly integrates the audit results of the text matching strategy and the image matching strategy, flexibly combines the audit results of the text matching strategy and the image matching strategy, and to a certain extent solves the image audit problem in complex customer standard scenarios. Specifically, the image audit according to the text-image combination strategy can be found in the description of steps 310 to 340 above, which will not be repeated here.

[0159] The multimodal model strategy fuses the image features of the picture and the text features of the text information in the picture to obtain a more accurate multimodal fusion feature, and further inputs the multimodal fusion feature into the classifier to obtain a more accurate picture review result. The multimodal model strategy is suitable for the review of pictures of some violation categories that are more difficult to review. Specifically, the picture review according to the multimodal model strategy can be found in the description of steps 350 to 380 above, which will not be repeated here.

[0160] Based on the above-mentioned single-mode strategy and multi-mode strategy, the embodiments of the present application can provide customers with content review capabilities in multiple categories (category 1 to category 8 in the figure) in each customer scenario. The categories involved may include, for example, advertising, pornography, illegal and irregular activities, underage identification, sexy or other categories, without limitation. The embodiments of the present application can help customers effectively identify prohibited content and avoid customer operational risks.

[0161] Therefore, the embodiment of the present application innovatively introduces a comprehensive review method based on a combination of image and text strategies in the image review scenario, introduces text information in the image to assist in image content review, and provides a variety of text analysis methods such as text vocabulary, text rules, and deep learning models, which can significantly improve the accuracy and recall of image review results. For example, in recent advertising delivery scenarios, 16% of false positives were reduced, and the overall accuracy rate was increased from 88.9% to 94.8%. In addition, by introducing and multimodal model strategies, text features and image features are fused based on multimodal models, which increases the recall rate by 21.6% in scale-type scenarios with greater review difficulty.

[0162] In addition, the effect optimization of existing solutions mostly relies on the update and iteration of image deep learning models, but the links involved in the optimization of image deep learning models are often involved, such as image sample collection, image sample annotation, image sample data enhancement, model training, model online and other steps, and the model optimization cycle is relatively long. The embodiment of the present application provides powerful text thesaurus, text rules and other strategies, and supports the configuration of graphic and text combination strategies, allowing for fast and flexible configuration according to specific customer scenarios, greatly reducing the response time for customer review needs. For example, the response cycle for customers to add new standards has been shortened from 2 weeks in the past to 2 days now.

[0163] The specific implementation modes of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above implementation modes. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific implementation modes can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different implementation modes of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0164] It should also be understood that in the various method embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. It should be understood that these sequence numbers can be interchanged where appropriate, so that the embodiments of the present application described can be implemented in an order other than those shown or described.

[0165] The above describes the method embodiment of the present application in detail. Figure 17 to Figure 18 , describe in detail the device embodiments of the present application.

[0166] Fig.17 FIG. 1 is a schematic block diagram of a device 10 for image review according to an embodiment of the present application. Fig.17 As shown, the device 10 may include a character recognition unit 11 , a text matching unit 12 , a picture matching unit 13 and a determination unit 14 .

[0167] The character recognition module 11 is used to detect and recognize characters in the image to be reviewed, and extract text information from the image to be reviewed;

[0168] A text matching unit 12 is used to match the text information with a preset text tag and determine a text matching result, wherein the text matching result is used to indicate whether the text information successfully matches the preset text tag; wherein the preset text tag includes at least one of at least one keyword pre-stored in a text word library, at least one pre-configured regular expression, and at least one text category;

[0169] The image matching unit 13 is used to match the image to be reviewed with a preset image tag and determine an image matching result, wherein the image matching result is used to indicate whether the image to be reviewed successfully matches the preset image tag; wherein the preset image tag includes at least one of at least one image sample feature pre-stored in the image library, at least one pre-configured image category, and at least one target image;

[0170] The determination unit 14 is used to determine whether the image to be reviewed is compliant according to the text matching result and the image matching result.

[0171] In some embodiments, the preset text label includes a text label for indicating that the target type information in the image is compliant.

[0172] In some embodiments, the text matching unit 12 is specifically used for:

[0173] The text information is input into a text classification model to determine a text category matching the text information in the at least one text category as the text matching result; wherein the at least one text category includes a label type for indicating text that complies with the target type information, a label type for indicating that the text information violates the regulations, and a label type for indicating that the text information complies.

[0174] In some embodiments, the preset image tag includes an image tag for indicating that the picture includes target type information.

[0175] In some embodiments, the picture matching unit 13 is specifically used for:

[0176] The image to be reviewed is input into an image similarity model, and similarity matching is performed between the image to be reviewed and the at least one image sample feature in the image library to obtain the image matching result; the at least one image sample feature includes image features of images containing the target type information and image features of images containing violation information.

[0177] In some embodiments, the picture matching unit 13 is specifically used for:

[0178] The image to be reviewed is input into an image classification model to determine an image type that matches the image to be reviewed in the at least one image category as the image matching result; wherein the at least one image category includes a label type for indicating that the image contains the target type information, a label type for indicating that the image contains violation information, and a label type for indicating that the image is compliant.

[0179] In some embodiments, the picture matching unit 13 is specifically used for:

[0180] The image to be reviewed is input into an image detection model, and the image to be reviewed is matched with the at least one target image to obtain the image matching result; wherein the at least one target image includes an image containing the target type information and an image containing violation information.

[0181] In some embodiments, when the preset text tag includes a text tag for indicating that the target type information in the picture is compliant, and the preset image tag includes an image tag for indicating that the picture includes the target type information, the determining unit 14 is specifically configured to:

[0182] When the text matching result is used to indicate that the text information successfully matches the preset text tag, and the image matching result is used to indicate that the image to be reviewed successfully matches the preset image tag, it is determined that the image to be reviewed is compliant.

[0183] In some embodiments, the determining unit 14 is specifically configured to:

[0184] When the text matching result is used to indicate that the text information does not successfully match the preset text tag, and the image matching result is used to indicate that the image to be reviewed successfully matches the preset image tag, it is determined that the image to be reviewed is non-compliant.

[0185] In some embodiments, it also includes:

[0186] A feature extraction unit is used to: extract features from the text information to obtain text features of the text information; and extract features from the image to be reviewed to obtain image features of the image to be reviewed;

[0187] A fusion unit, used for inputting the text features and the image features into a neural network model, and fusing the text features and the image features to obtain a multimodal fusion feature;

[0188] A classification unit is used to input the multimodal fusion feature into a classifier to obtain a classification label of the image to be reviewed, and the classification label is used to indicate whether the image to be reviewed is compliant.

[0189] In some embodiments, the text vocabulary also includes at least one keyword for indicating violation information.

[0190] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.17 The device 10 shown can execute the above method embodiment, and the above and other operations and / or functions of each module in the device 10 are respectively for implementing the corresponding processes in the above method 300, which will not be repeated here for the sake of brevity.

[0191] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0192] Fig.18 It is a schematic block diagram of an electronic device 30 provided in an embodiment of the present application.

[0193] like Fig.18 As shown, the electronic device 30 may include:

[0194] The memory 31 and the processor 32, the memory 31 is used to store the computer program and transmit the program code to the processor 32. In other words, the processor 32 can call and run the computer program from the memory 31 to implement the method in the embodiment of the present application.

[0195] For example, the processor 32 may be configured to execute the above method embodiments according to instructions in the computer program.

[0196] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0197] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0198] In some embodiments of the present application, the memory 31 includes but is not limited to:

[0199] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0200] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0201] like Fig.18 As shown, the electronic device 30 may further include:

[0202] The transceiver 33 may be connected to the processor 32 or the memory 31 .

[0203] The processor 32 may control the transceiver 33 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 730 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.

[0204] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0205] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0206] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0207] It is understandable that in the specific implementation of the present application, when the above embodiments of the present application are applied to specific products or technologies and involve relevant data such as user information, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards.

[0208] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0210] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0211] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for image review, characterized in that: include: Using a character recognition module to perform text detection and recognition on the image to be reviewed, and extracting text information from the image to be reviewed; Matching the text information with a preset text tag to determine a text matching result, wherein the text matching result is used to indicate whether the text information successfully matches the preset text tag; wherein the preset text tag includes at least one of at least one keyword pre-stored in a text vocabulary, at least one pre-configured regular expression, and at least one text category; Matching the image to be reviewed with a preset image tag to determine an image matching result, wherein the image matching result is used to indicate whether the image to be reviewed successfully matches the preset image tag; wherein the preset image tag includes at least one of at least one image sample feature pre-stored in an image library, at least one pre-configured image category, and at least one target image; Whether the image to be reviewed is compliant is determined based on the text matching result and the image matching result.

2. The method according to claim 1, characterized in that The preset text label includes a text label used to indicate that the target type information in the image is compliant.

3. The method according to claim 2, characterized in that The matching of the text information with a preset text tag to determine a text matching result includes: The text information is input into a text classification model to determine a text category matching the text information in the at least one text category as the text matching result; wherein the at least one text category includes a label type for indicating text that complies with the target type information, a label type for indicating that the text information violates the regulations, and a label type for indicating that the text information complies.

4. The method according to claim 1, characterized in that: The preset image tag includes an image tag for indicating that the picture includes target type information.

5. The method according to claim 4, characterized in that Matching the image to be reviewed with a preset image tag to determine an image matching result includes: The image to be reviewed is input into an image similarity model, and similarity matching is performed between the image to be reviewed and the at least one image sample feature in the image library to obtain the image matching result; the at least one image sample feature includes image features of images containing the target type information and image features of images containing violation information.

6. The method according to claim 4, characterized in that Matching the image to be reviewed with a preset image tag to determine an image matching result includes: The image to be reviewed is input into an image classification model to determine an image type that matches the image to be reviewed in the at least one image category as the image matching result; wherein the at least one image category includes a label type for indicating that the image contains the target type information, a label type for indicating that the image contains violation information, and a label type for indicating that the image is compliant.

7. The method according to claim 4, characterized in that The step of matching the image to be reviewed with a preset image tag to determine an image matching result includes: The image to be reviewed is input into an image detection model, and the image to be reviewed is matched with the at least one target image to obtain the image matching result; wherein the at least one target image includes an image containing the target type information and an image containing violation information.

8. The method according to claim 1, characterized in that When the preset text tag includes a text tag for indicating that the target type information in the image is compliant, and the preset image tag includes an image tag for indicating that the image includes the target type information, determining whether the image to be reviewed is compliant according to the text matching result and the image matching result includes: When the text matching result is used to indicate that the text information successfully matches the preset text tag, and the image matching result is used to indicate that the image to be reviewed successfully matches the preset image tag, it is determined that the image to be reviewed is compliant.

9. The method according to claim 8, characterized in that Also includes: When the text matching result is used to indicate that the text information does not successfully match the preset text tag, and the image matching result is used to indicate that the image to be reviewed successfully matches the preset image tag, it is determined that the image to be reviewed is non-compliant.

10. The method according to claim 1, characterized in that Also includes: Performing feature extraction on the text information to obtain text features of the text information; Extracting features of the image to be reviewed to obtain image features of the image to be reviewed; Inputting the text features and the image features into a neural network model, and fusing the text features and the image features to obtain a multimodal fusion feature; The multimodal fusion feature is input into a classifier to obtain a classification label of the image to be reviewed, and the classification label is used to indicate whether the image to be reviewed is compliant.

11. The method according to any one of claims 1 to 10, characterized in that: The text word library also includes at least one keyword for indicating violation information.

12. A device for image review, characterized in that: include: A character recognition module is used to detect and recognize characters in the image to be reviewed, and extract text information from the image to be reviewed; A text matching unit, used to match the text information with a preset text tag and determine a text matching result, wherein the text matching result is used to indicate whether the text information successfully matches the preset text tag; wherein the preset text tag includes at least one of at least one keyword pre-stored in a text word library, at least one pre-configured regular expression, and at least one text category; An image matching unit is used to match the image to be reviewed with a preset image tag and determine an image matching result, wherein the image matching result is used to indicate whether the image to be reviewed successfully matches the preset image tag; wherein the preset image tag includes at least one of at least one image sample feature pre-stored in an image library, at least one pre-configured image category, and at least one target image; A determination unit is used to determine whether the image to be reviewed is compliant based on the text matching result and the image matching result.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, the processor executes the method according to any one of claims 1 to 11.

14. A computer storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for executing the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The method comprises a computer program code, and when the computer program code is executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 11.

Images