Abnormality detection method and device, electronic equipment and storage medium

Abstract

The invention provides an anomaly detection method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the fields of large models, generative models, multi-modal models, text processing, image processing and the like. According to the specific implementation scheme, in response to acquired input information, target detection is carried out on an input image in the input information, and the position and the abnormality category of an abnormal area in the input image are obtained; and generating an anomaly detection result for the input information based on the large model according to the input text, the input image, the position of the abnormal region, the anomaly category and the target reference information in the input information.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of large models, generative models, multimodal models, text processing, image processing, etc. More specifically, the present disclosure provides an anomaly detection method, device, electronic device, storage medium, and computer program product. Background Art

[0002] Some Internet platforms need to review the information uploaded by users and take some countermeasures if the information contains inappropriate content to ensure the security of platform information. Summary of the invention

[0003] The present disclosure provides an anomaly detection method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, there is provided an anomaly detection method, comprising: in response to obtaining input information, performing target detection on an input image in the input information to obtain a position of an abnormal area and an abnormal category in the input image; and generating an anomaly detection result for the input information based on a large model according to input text, input image, position of the abnormal area, abnormal category and target reference information in the input information.

[0005] According to another aspect of the present disclosure, an abnormality detection device is provided, comprising: a target detection module and a generation module. The target detection module is used to perform target detection on an input image in the input information in response to obtaining input information, and obtain the position and abnormality category of the abnormal area in the input image. The generation module is used to generate an abnormality detection result for the input information based on a large model according to the input text, input image, position of the abnormal area, abnormality category and target reference information in the input information.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an application scenario of the anomaly detection method and device according to an embodiment of the present disclosure;

[0012] Figure 2 is a schematic flow chart of an anomaly detection method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic flow chart of an anomaly detection method according to another embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of an abnormality detection method according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic structural block diagram of an abnormality detection device according to an embodiment of the present disclosure; and

[0016] Figure 6 It is a structural block diagram of an electronic device used to implement the abnormality detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0019] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0020] Some Internet platforms need to review information uploaded by users. For example, when users upload text or images, they need to review them to confirm whether the information contains violent, sensitive or other illegal and negative information.

[0021] In some related technologies, for example, specific categories of bad information can be detected through preset rules or models. However, rule-making is difficult to cope with the diversity of content and there is a high false alarm rate. For another example, deep neural networks can be used for image classification or target detection, but the accuracy of this detection method is low, and it is easy to intercept normal content, affecting the user experience. For another example, a preliminary review can be conducted through a deep neural network, and then the user experience can be enhanced through manual review. However, the real-time performance of this method is low, and it is difficult to meet the security and real-time requirements in large model scenarios. In summary, the review methods provided by related technologies have the problems of low accuracy and low real-time performance.

[0022] The disclosed embodiment aims to provide a detection method, which first obtains input information, including input image and input text. Then, target detection is performed on the input image to preliminarily confirm the location of the abnormal area and the abnormal category. Then, target reference information of the same field is retrieved based on the input text. Then, a large model is used to process the input text, the input image, the location of the abnormal area, the abnormal category and the target reference information to generate an abnormality detection result.

[0023] It can be seen that the above scheme can improve the accuracy of anomaly detection results by auditing the target detection and the large model separately. In addition, since automatic detection can be performed without manual audit, the real-time audit can be ensured.

[0024] The technical solution provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Figure 1 It is a schematic diagram of an application scenario of the anomaly detection method and device according to an embodiment of the present disclosure.

[0026] It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0029] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as determining an abnormality detection result based on the user's input information, and generating prompt information as the processing result when an abnormality is determined, and the abnormality detection result and prompt information may be used as the processing result) to the terminal device.

[0030] It should be noted that the anomaly detection method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the anomaly detection device provided in the embodiment of the present disclosure can generally be set in the server 105. The anomaly detection method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the anomaly detection device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0031] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0032] Figure 2 is a schematic flow chart of an anomaly detection method according to an embodiment of the present disclosure.

[0033] like Figure 2 As shown, the abnormality detection method 200 may include operations S210 to S220.

[0034] In operation S210 , in response to obtaining input information, target detection is performed on an input image in the input information to obtain a position and an abnormality category of an abnormal region in the input image.

[0035] In one application scenario, the input information may be input by the user through the front-end page, and the input text may be a story or joke written by the user, or a question raised by the user. In this application scenario, the anomaly detection method in this example can be used to detect whether the information uploaded by the user is abnormal.

[0036] In another application scenario, the input information may be generated by a generative model. For example, the generative model generates graphic information, and the graphic information may be detected to determine whether the information generated by the generative model is abnormal.

[0037] For example, the input information includes input text and input images. You can use YOLO (You Only Look Once, a target detection model) or other algorithms to perform target detection on the input image to determine whether there are abnormal areas in the input image, and determine the location and abnormal category of the abnormal area if there are abnormal areas. The above-mentioned abnormal area may contain illegal content, and the abnormal category includes, for example, infringement of the legitimate rights and interests of others, bad language, etc. For example, the input image includes a flag, and the flag contains some bad language. The position of the flag in the input image can be determined through target detection, and it can be determined that the abnormal category of the flag is "bad language".

[0038] In operation S220, an abnormality detection result for the input information is generated based on the large model according to the input text, the input image, the location of the abnormal area, the abnormality category, and the target reference information in the input information.

[0039] For example, the large model may be a large language model, a multimodal large model, etc. This embodiment does not limit the large model.

[0040] For example, the input text, the location of the abnormal area, the abnormal category, the target reference information and the prompt information can be combined, and then the combined text and the input image are input into the large model, and the large model generates the abnormality detection result. For example, the prompt information is a natural language text, and the prompt information is such as "The area from the upper left corner coordinate (X1, Y1) to the lower right corner coordinate (X2, Y2) in the image is suspected to be illegal, and the violation category is XX. Please determine whether it is illegal and explain the reason", where the violation category in the prompt information is abnormal and has not changed, and the coordinates (X1, Y1) and coordinates (X2, Y2) are determined based on the location of the abnormal area.

[0041] The disclosed embodiment preliminarily determines whether the input image is abnormal through target detection, and then, if it is determined that the input image is abnormal, performs a secondary review through the large model to determine whether the input information is abnormal, thereby improving the accuracy of the abnormality detection result. In addition, since automatic detection is possible without manual review, the real-time review can be ensured.

[0042] Figure 3 is a schematic flowchart of an anomaly detection method according to another embodiment of the present disclosure.

[0043] like Figure 3 As shown, the abnormality detection method 300 may include operation S310, operation S315~operation S316 and operation S320.

[0044] In operation S310 , in response to obtaining input information, target detection is performed on an input image in the input information to obtain a position and an abnormality category of an abnormal region in the input image.

[0045] In operation S315, according to the abnormal area, it is determined whether there is target data matching the abnormal area in the first database, wherein the data stored in the first database is determined based on the abnormal information. If yes, operation S316 is executed, if not, operation S320 is executed.

[0046] For example, the first database may be constructed in advance based on the abnormal information. The abnormal information may be data in a modality such as text, image, video, etc., and the abnormal information may contain abnormal content such as bad language. The abnormal information may be added to the first database, or feature extraction may be performed on the abnormal information, and then the extracted features may be added to the first database.

[0047] For example, a sub-image of the abnormal area can be cropped from the input image according to the position of the abnormal area. Then, the features of the sub-image can be extracted. The similarity between the features of the sub-image and the data in the first database can also be determined, and it can be determined whether the similarity is greater than or equal to a similarity threshold. If so, it can be determined that the data matches the abnormal area and the data is determined as the target data. If not, it is determined that the data does not match the abnormal area and the data is determined not to be the target data.

[0048] For another example, the category of abnormal information can be detected and recorded through a pre-trained classification model. If the similarity between a certain data in the first database and the abnormal region in the input image is greater than a threshold, and the category of the data is consistent with the abnormal category of the input image, the data is determined to match the abnormal region and the data is determined to be the target data. If not, the data is determined not to match the abnormal region and the data is determined not to be the target data.

[0049] In operation S316, determining the abnormality detection result includes: the input information is abnormal. In addition, the abnormality detection result may also include an abnormal category of the input information, and the abnormal category of the input information is consistent with the category of the matched target data.

[0050] In operation S320, an abnormality detection result for the input information is generated based on the large model according to the input text, the input image, the location of the abnormal area, the abnormality category, and the target reference information in the input information.

[0051] In this embodiment, the first database is first searched to see whether there is target data matching the abnormal area. If so, the input information is directly judged to be abnormal. If not, a second judgment is made through the large model. This can avoid all input information being tested through the large model, reduce the number of calls to the large model, and reduce the cost of using the large model.

[0052] According to another embodiment of the present disclosure, the above-mentioned object detection on the input image in the input information may include: determining the position and category confidence of the region of interest in the input image. Further, according to the detection scene, determining the target threshold. Then, in response to detecting that the category confidence is greater than or equal to the target threshold, determining the region of interest as an abnormal region.

[0053] For example, the location and confidence category of the region of interest can be determined by a target detection model, which can be YOLO or other models, such as PPYOLOE or other lightweight models suitable for fast object detection in a real-time environment. This embodiment does not limit the target detection model. The target detection model needs to be trained in advance. During the training process, self-supervised training can be performed through a large-scale first data set, and supervised training can also be performed through a second data set. The data in the second data set may include data generated based on artificial intelligence, so that the target detection model can identify whether the information generated by artificial intelligence is abnormal.

[0054] For example, the correspondence between the detection scene and the target threshold may be preconfigured, and then the target threshold required for the current scene may be selected based on the correspondence, and then whether the region of interest is an abnormal region may be determined according to the target threshold.

[0055] This embodiment automatically adjusts the target threshold according to the detection scenario. Different detection scenarios may have different requirements for target detection. By setting an appropriate target threshold, it can be ensured that the target detection result meets the requirements of the scenario, thereby improving the accuracy of the detection result.

[0056] In the actual detection process, the target threshold of target detection can be set relatively low to avoid missed detection. At the same time, the input information of false detection can be re-detected through the large model to ensure the accuracy of detection.

[0057] According to another embodiment of the present disclosure, the input of the large model may further include target reference information. Next, the process of determining the target reference information is described.

[0058] In this embodiment, the candidate reference information can be retrieved in the second database based on the input text based on the predetermined search rules. Then the input text and the candidate reference information are classified to obtain the fields involved in the input text and the fields involved in the candidate reference information. If the fields involved in the candidate reference information are the same as the fields involved in the input text, the candidate reference information is determined as the target reference information.

[0059] For example, a second database may be pre-constructed, and the reference information stored in the second database may include reference information with high confidence in multiple fields. For example, the confidence of the reference information is greater than a threshold, indicating that the reference information is basically correct information based on facts rather than erroneous information.

[0060] For example, the predetermined search rule is pre-configured according to actual needs, such as a search based on text similarity, a search based on semantic similarity, etc., which is not limited in this embodiment.

[0061] For example, it can be determined whether the input text meets the conversion conditions. If the input text meets the conversion conditions, the input text can be subjected to text conversion processing to obtain the converted text. In one example, the conversion conditions include, for example: the number of characters in the input text is greater than or equal to a first quantity threshold, and accordingly, the text conversion processing may include extracting summaries, extracting keywords, etc. In one example, the conversion conditions include, for example: the number of characters in the input text is less than or equal to a second quantity threshold, and the second quantity threshold is less than the first quantity threshold, and accordingly, the text conversion processing may include expanding the input text. If the number of characters in the input text is between the first quantity threshold and the second quantity threshold, text conversion may not be performed on the input text.

[0062] The retrieval may be performed based on the converted text, for example, by calculating the similarity between the converted text and each reference information in the second database, and when the similarity is greater than a threshold, the reference information is determined as candidate reference information.

[0063] Next, the pre-trained classification model can be used to detect the fields involved in the input text and the fields involved in the candidate reference information. If the fields involved in the candidate reference information are the same as the fields involved in the input text, the candidate reference information is determined as the target reference information, otherwise the candidate reference information is not determined as the target reference information.

[0064] It should be noted that, in other embodiments, the input text may not be converted, but the input text before conversion may be directly used to search in the second database.

[0065] In this embodiment, candidate reference information related to the input text is first retrieved from the second database, and then the candidate reference information is filtered by domain to obtain the target reference information. Then, the target reference information is input into the large model, and the target reference information is used to provide more information for the large model, thereby assisting in the generation of the target detection result. This can alleviate the hallucination problem of the large model, thereby further improving the accuracy of the detection result.

[0066] For example, the input text includes "Zhang San". The first candidate reference information obtained after retrieval includes: the personal introduction of a sports star named Zhang San, and the second candidate reference information also includes: the personal introduction of a doctor named Zhang San. After filtering by domain, the first candidate reference information is filtered out, and the second candidate reference information is retained. In this way, the large model can generate an anomaly detection result based on the second candidate reference information, thereby avoiding answering based on content related to the sports field.

[0067] Figure 4 It is a schematic principle diagram of the anomaly detection method according to an embodiment of the present disclosure.

[0068] In this embodiment, input information is first obtained, and the input information includes an input image 401 and an input text 406.

[0069] For the input image 401 in the input information, the location 403 and anomaly category 402 of the anomaly area can be determined through an object detection model. Then, according to the location 403 of the anomaly area, a sub-image 404 of the anomaly area is determined from the input image 401, and the similarity between the sub-image 404 and each data in the first database 405 is determined.

[0070] If the similarity between a certain data and the sub-image 404 is greater than or equal to the similarity threshold, it is determined that the data matches the sub-image 404, and the data is determined as the target data. At this time, it can be determined that the input information is abnormal, which is used as the anomaly detection result.

[0071] If the similarity between each data in the first database 405 and the sub-image 404 is less than the similarity threshold, it is determined that the first database 405 lacks target data that matches the sub-image 404. Next, based on a predetermined retrieval rule, candidate reference information can be retrieved from the second database 407 according to the input text 406. Then, the input text 406 and the candidate reference information are classified to obtain the field involved in the input text 406 and the field involved in the candidate reference information.

[0072] If the field involved in a certain candidate reference information is the same as the field involved in the input text 406, the candidate reference information is determined as the target reference information 408, otherwise the candidate reference information is filtered out, and in this way, the target reference information 408 can be obtained.

[0073] Next, the input text 406, the location of the abnormal area 403, the abnormal category 402, the target reference information 408 and the prompt information can be combined, and then the combined information and the input image 401 are input into the large model 409, and the large model 409 generates the abnormality detection result 410.

[0074] In this embodiment, target detection is first used to determine whether the input image 401 has an abnormality. Then, when the target detection model detects an abnormal area, a secondary confirmation is performed by matching the input image 401 with the data in the first database 405, thereby reducing false detections. If the target detection result indicates that there is an abnormal area in the input image 401, and the target data is not matched in the first database 405, the large model 409 is used to review it again to confirm whether the input information is abnormal. In addition, in the process of generating the abnormal detection result 410 based on the large model 409, the target reference information 408 is also provided to the large model 409 to alleviate the hallucination problem of the large model 409 and further improve the accuracy of the detection result. Therefore, the abnormal detection method provided in this embodiment combines target detection, feature matching and the large model 409, which can improve detection accuracy, reduce false alarm rate, and improve real-time performance, so as to achieve efficient and real-time content review.

[0075] Figure 5 It is a schematic structural block diagram of an abnormality detection device according to an embodiment of the present disclosure.

[0076] like Figure 5 As shown, the anomaly detection device 500 may include a target detection module 510 and a generation module 520 .

[0077] The target detection module 510 is used to perform target detection on the input image in the input information in response to obtaining the input information, and obtain the position and abnormality category of the abnormal area in the input image.

[0078] The generation module 520 is used to generate anomaly detection results for the input information based on the large model according to the input text, input image, location of the abnormal area, abnormal category and target reference information in the input information.

[0079] According to another embodiment of the present disclosure, the above-mentioned device further includes: a determination module and a result determination module. The determination module is used to determine whether there is target data matching the abnormal area in the first database according to the abnormal area; wherein the data stored in the first database is determined based on the abnormal information. The result determination module is used to determine, in response to detecting that the target data exists in the first database, that the abnormality detection result includes: the input information is abnormal.

[0080] According to another embodiment of the present disclosure, the determination module includes: a sub-image determination sub-module, a similarity determination sub-module and a target data determination sub-module. The sub-image determination sub-module is used to determine the sub-image of the abnormal area from the input image according to the position of the abnormal area. The similarity determination sub-module is used to determine the similarity between the sub-image and the data in the first database. The target data determination sub-module is used to determine the data used to determine the similarity as the target data in response to detecting that the similarity is greater than or equal to the similarity threshold.

[0081] According to another embodiment of the present disclosure, the above-mentioned device also includes: a trigger module, which is used to trigger the operation of generating anomaly detection results for input information based on the large model in response to detecting that the target data is missing in the first database.

[0082] According to another embodiment of the present disclosure, the target detection module includes: a first determination submodule, a second determination submodule and a third determination submodule. The first determination submodule is used to determine the position and category confidence of the region of interest in the input image. The second determination submodule is used to determine the target threshold according to the detection scene. The third determination submodule is used to determine the region of interest as an abnormal region in response to detecting that the category confidence is greater than or equal to the target threshold.

[0083] According to another embodiment of the present disclosure, the above-mentioned device also includes: a retrieval module, a classification module and a reference information determination module. The retrieval module is used to retrieve candidate reference information in the second database according to the input text based on a predetermined retrieval rule. The classification module is used to classify the input text and the candidate reference information to obtain the field involved in the input text and the field involved in the candidate reference information. The reference information determination module is used to determine the candidate reference information as the target reference information in response to detecting that the field involved in the candidate reference information is the same as the field involved in the input text.

[0084] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned abnormality detection method.

[0085] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above-mentioned abnormality detection method.

[0086] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, and the computer program implements the above-mentioned abnormality detection method when executed by a processor.

[0087] Figure 61 is a block diagram of an electronic device for implementing the anomaly detection method of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0088] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0089] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0090] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the anomaly detection method. For example, in some embodiments, the anomaly detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the anomaly detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the anomaly detection method in any other appropriate manner (e.g., by means of firmware).

[0091] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0093] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0095] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0096] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0097] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0098] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An anomaly detection method, comprising: In response to obtaining the input information, performing target detection on the input image in the input information to obtain the position and abnormality category of the abnormal area in the input image; as well as According to the input text in the input information, the input image, the position of the abnormal area, the abnormal category and the target reference information, an abnormality detection result for the input information is generated based on the large model.

2. The method according to claim 1, further comprising: According to the abnormal area, determining whether there is target data matching the abnormal area in the first database; wherein the data stored in the first database is determined based on the abnormal information; and In response to detecting that the target data exists in the first database, it is determined that the input information is abnormal.

3. The method according to claim 2, wherein: Determining, according to the abnormal area, whether there is target data matching the abnormal area in the first database includes: Determining a sub-image of the abnormal area from the input image according to the position of the abnormal area; Determine the similarity between each data in the first database and the sub-image; and In response to detecting that the similarity between at least one data in the first database and the sub-image is greater than or equal to a similarity threshold, the at least one data is determined as the target data.

4. The method according to claim 2, further comprising: In response to detecting that the target data is missing from the first database, an operation of generating an anomaly detection result for the input information based on the large model is triggered.

5. The method according to claim 1, performing target detection on the input image in the input information comprises: Determining a location and a class confidence of a region of interest in the input image; Determine the target threshold according to the detection scenario; as well as In response to detecting that the category confidence is greater than or equal to the target threshold, the region of interest is determined as the abnormal region.

6. The method according to claim 1, further comprising: Based on a predetermined search rule, searching for candidate reference information in a second database according to the input text; Classifying the input text and the candidate reference information to obtain the fields involved in the input text and the fields involved in the candidate reference information; as well as In response to detecting that the field involved in the candidate reference information is the same as the field involved in the input text, the candidate reference information is determined as the target reference information.

7. An abnormality detection device, comprising: A target detection module, configured to, in response to obtaining input information, perform target detection on an input image in the input information to obtain a position and an abnormality category of an abnormal area in the input image; as well as A generation module is used to generate an anomaly detection result for the input information based on a large model according to the input text in the input information, the input image, the position of the abnormal area, the abnormal category and the target reference information.

8. The apparatus according to claim 7, further comprising: A determination module, configured to determine, based on the abnormal area, whether there is target data matching the abnormal area in the first database; wherein the data stored in the first database is determined based on the abnormal information; and A result determination module is configured to determine that the input information is abnormal in response to detecting that the target data exists in the first database.

9. The device according to claim 8, wherein: The determination module includes: A sub-image determination submodule, used to determine a sub-image of the abnormal area from the input image according to the position of the abnormal area; a similarity determination submodule, configured to determine the similarity between the sub-image and each data in the first database; and The target data determination submodule is configured to determine the at least one data in the first database as the target data in response to detecting that the similarity between the at least one data in the first database and the sub-image is greater than or equal to a similarity threshold.

10. The apparatus according to claim 8, further comprising: A trigger module is used to trigger the operation of generating an anomaly detection result for the input information based on the large model in response to detecting that the target data is missing in the first database.

11. The device according to claim 7, wherein the target detection module comprises: A first determination submodule, used to determine the position and category confidence of the region of interest in the input image; The second determination submodule is used to determine the target threshold according to the detection scenario; as well as The third determination submodule is configured to determine the region of interest as the abnormal region in response to detecting that the category confidence is greater than or equal to the target threshold.

12. The apparatus according to claim 7, further comprising: A retrieval module, configured to retrieve candidate reference information in a second database based on a predetermined retrieval rule and according to the input text; A classification module, used to classify the input text and the candidate reference information to obtain the fields involved in the input text and the fields involved in the candidate reference information; as well as The reference information determination module is configured to determine the candidate reference information as the target reference information in response to detecting that the field involved in the candidate reference information is the same as the field involved in the input text.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images