Method and apparatus for detecting image anomalies

Abstract

This invention provides a method and apparatus for detecting image anomalies that use both visual and textual features contained in an image to detect abnormal signs in the image. [Solution] A method for detecting image anomalies according to one embodiment of the present invention includes the steps of: collecting at least one image information and dividing the at least one image information into a plurality of snippets; extracting visual feature information and text feature information for each of the divided plurality of snippets; synthesizing the extracted visual feature information and text feature information to generate composite information; and detecting signs of anomalies in at least one image information based on the generated composite information.

Description

[Technical Field]

[0001] The present invention relates to an image anomaly detection method and apparatus capable of detecting abnormal signs contained in an image. [Background technology]

[0002] Recently, image anomaly detection, which allows for the identification of abnormal data contained in images without direct human observation, has been applied and used in a variety of fields. For example, in the manufacturing sector, it can detect abnormal behavior and irregular work in the production process, and in the medical field, it can monitor the patient's condition and automatically send an alarm to medical staff if an accident occurs. In the public safety sector, it has been used to protect human life and property by recognizing illegal acts or accidental actions in images in advance.

[0003] For image anomaly detection to proceed smoothly, a good balance of normal and abnormal data is necessary. However, in daily operations, the occurrence of events corresponding to abnormal data is infrequent, leading to a shortage of abnormal data and an imbalance between normal and abnormal data, which makes smooth image anomaly detection difficult. Furthermore, the lack of clear criteria for distinguishing between normal and abnormal data also hinders smooth image anomaly detection.

[0004] Traditionally, to address this problem, image anomaly detection was performed based on information about the visual features contained in the image, depending on the degree of supervision involvement of the image observer. However, relying solely on information about visual features makes it difficult to clearly determine the semantic meaning contained in the image, frequently leading to a decrease in the accuracy of image anomaly detection.

[0005] The matters described above as background technology are intended solely to enhance understanding of the background of the present invention and should not be taken as constituting prior art already known to those with ordinary skill in the art. [Overview of the project] [Problems that the invention aims to solve]

[0006] The present invention is proposed to solve the aforementioned problems, and its objective is to provide a method and apparatus for detecting image anomalies that use not only visual features but also text features to detect abnormal signs contained in an image. The technical problems that the present invention aims to solve are not limited to the technical problems described above, and other technical problems not mentioned above will be clearly understood by those with ordinary skill in the art to which the present invention belongs from the following description. [Means for solving the problem]

[0007] A method for detecting image anomalies according to the present invention to achieve the above objective may include the steps of: collecting at least one image information and dividing the at least one image information into a plurality of snippets; extracting visual feature information and text feature information for each of the divided plurality of snippets; synthesizing the extracted visual feature information and text feature information to generate composite information; and detecting signs of anomalies in the at least one image information based on the generated composite information.

[0008] For example, the extraction step may include a step of performing a preprocessing on each of the divided snippets, such that the frame is cropped according to a predetermined number of cropping steps, and a step of extracting the visual feature information for each of the preprocessed snippets.

[0009] For example, the extraction step may include the step of generating captions for each of the divided snippets, and the step of extracting the text feature information by performing text embedding based on the generated captions.

[0010] For example, the extraction step may include a step of applying multi-scale correction to the visual feature information and text feature information for each of the multiple snippets, and a step of extracting the corrected visual feature information and text feature information.

[0011] For example, the generation step may include generating the synthesized information using at least one of a plurality of synthesis methods based on the extracted visual feature information and text feature information.

[0012] For example, the sensing step may include the steps of determining a first loss value corresponding to the total loss value of the at least one image information and a second loss value corresponding to a learned value for sensing an abnormality in the at least one image information, based on the generated composite information; generating a loss function based on the determined first loss value and second loss value; and sensing an abnormality in the at least one image information based on the generated loss function.

[0013] For example, the step of determining the first loss value and the second loss value may include the steps of determining the feature size corresponding to each of the plurality of snippets based on the generated composite information, determining the average feature size of the feature sizes that correspond to a predetermined number of feature sizes in order of size from among the determined feature sizes, and determining the first loss value based on the determined average feature size.

[0014] For example, the steps for determining the first loss value and the second loss value may include: determining the feature size corresponding to each of the plurality of snippets based on the generated composite information; determining the frame level prediction value for each of the plurality of snippets based on the determined feature size; determining the number of anomalies in at least one image information based on the frame level prediction values ​​corresponding to a predetermined number of feature sizes in order of size from among the feature sizes; and determining the second loss value based on the determined number of anomalies.

[0015] For example, after the sensing step, if the abnormality signs are detected, the process may further include outputting the number of abnormal points in the at least one image information and the caption included in the at least one image information.

[0016] For example, the output step may include a step of outputting a warning alarm if the number of abnormal points in at least one image information exceeds a predetermined threshold.

[0017] Furthermore, the present invention for detecting image anomalies to achieve the above objective may include: an image acquisition unit that collects at least one image information and divides the at least one image information into a plurality of snippets; an information extraction unit that extracts visual feature information and text feature information for each of the plurality of snippets divided by the image acquisition unit; and a sensing control unit that synthesizes the extracted visual feature information and text feature information to generate synthesized information and senses signs of anomalies in the at least one image information based on the generated synthesized information.

[0018] For example, the information extraction unit may perform preprocessing on each of the divided snippets, cropping the frame according to a predetermined number of cropping cycles, and then extract the visual feature information for each of the preprocessed snippets.

[0019] For example, the information extraction unit may generate captions included in each of the plurality of divided snippets, and perform sentence embedding based on the generated captions to extract the text feature information.

[0020] For example, the information extraction unit may perform correction on the visual feature information and the text feature information for each of the plurality of snippets at multiple scales, and extract the corrected visual feature information and text feature information.

[0021] For example, the perception control unit may generate the composite information using at least one of a plurality of composite methods based on the extracted visual feature information and text feature information.

[0022] For example, the perception control unit respectively determines a first loss value corresponding to the total loss value of the at least one image information and a second loss value corresponding to a learning value for detecting an abnormal sign of the at least one image information based on the generated composite information, generates a loss function based on the determined first loss value and second loss value, and may detect an abnormal sign of the at least one image information based on the generated loss function.

[0023] For example, the perception control unit determines a feature size corresponding to each of the plurality of snippets based on the generated composite information, determines an average feature size of the feature sizes corresponding to a predetermined reference number in the order of size among the determined feature sizes, and may determine the first loss value based on the determined average feature size.

[0024] For example, the sensing control unit may determine a feature size corresponding to each of the plurality of snippets based on the generated composite information, determine a frame level prediction value for each of the plurality of snippets based on the determined feature size, and determine an anomaly score of the at least one image information based on the frame level prediction value corresponding to a feature size that corresponds to a predetermined number of criteria in the order of size among the feature sizes, and determine the second loss value based on the determined anomaly score.

[0025] For example, when the anomaly sign is sensed, the sensing control unit may output an anomaly score of the at least one image information and a caption included in the at least one image information.

[0026] For example, when the anomaly score of the at least one image information exceeds a predetermined reference score, the sensing control unit may output a warning alarm.

Advantages of the Invention

[0027] According to the above, the method and apparatus for sensing image anomalies of the present invention can improve the accuracy in sensing image anomalies by sensing anomaly signs included in an image based on visual features and text features extracted from the image. Further, by performing correction with multiple scales for each of the visual features and text features, the robustness of the method and apparatus for sensing image anomalies can be improved. Further, by outputting an anomaly score for an image and a caption included in the image, a user using the method and apparatus for sensing image anomalies according to the present invention can efficiently respond to anomaly signs included in the image. Further, since not only visual features but also text features are used together to capture the semantic meaning of anomaly signs included in an image, it can be applied to various fields. The effects obtained by the present invention are not limited to the above-described effects, and other effects not described above will be clearly understood by those having ordinary knowledge in the technical field to which the present invention pertains from the following description. [Brief explanation of the drawing]

[0028] [Figure 1] This is a block diagram illustrating the configuration of a device for detecting image anomalies according to one embodiment of the present invention. [Figure 2] This is a block diagram illustrating the configuration of a device for detecting image anomalies according to one embodiment of the present invention. [Figure 3] This is a block diagram illustrating the configuration of a device for detecting image anomalies according to one embodiment of the present invention. [Figure 4] This is a block diagram illustrating the configuration of a device for detecting image anomalies according to one embodiment of the present invention. [Figure 5] This is a flowchart illustrating a method for detecting image anomalies according to one embodiment of the present invention. [Modes for carrying out the invention]

[0029] In describing the embodiments disclosed herein, if a specific description of a relevant known technology is deemed likely to disrupt the gist of the embodiments disclosed herein, such detailed description will be omitted. Furthermore, the accompanying drawings are merely for the purpose of facilitating the understanding of the embodiments disclosed herein, and should be understood not as limiting the technical ideas disclosed herein, but as including any modifications, equivalents, or substitutions that fall within the concept and technical scope of the present invention.

[0030] Terms including ordinal numbers such as "first," "second," etc., can be used to describe a variety of components, but these components are not limited by these terms. These terms are used solely for the purpose of distinguishing one component from another. When one component is said to be "linked" or "connected" to another component, it should be understood that this includes cases where it is directly linked or connected to the other component, as well as cases where another component is interposed between them. Conversely, when one component is said to be "directly linked" or "directly connected" to another component, it should be understood that there is no other component interposed between them.

[0031] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” are intended to specify the presence of features, figures, stages, operations, components, parts, or combinations thereof as described in the specification, and should be understood not to preemptively exclude the possibility of the presence or addition of one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0032] The embodiments disclosed herein will now be described in detail with reference to the attached drawings, but identical or similar components will be given the same reference numerals, regardless of whether they are reference numerals in the drawings, and redundant descriptions thereof will be omitted.

[0033] An embodiment of the present invention for detecting image anomalies will be described with reference to Figures 1 to 4. Figures 1 to 4 are block diagrams illustrating the configuration of an image anomaly detection device according to one embodiment of the present invention.

[0034] Referring to Figure 1, the image anomaly detection device according to an embodiment of the present invention can include an image acquisition unit 100, an information extraction unit 200, and a detection control unit 300. Figure 1 mainly shows the components according to an embodiment of the present invention, and it goes without saying that when actually realizing an image anomaly detection device, fewer or more components can be included. On the other hand, each of the components 100, 200, and 300 of the image anomaly detection device according to an embodiment of the present invention can include a communication device that communicates with other components, a memory that stores an operating system, logic instructions, input / output information, etc., and one or more processors that perform judgments, calculations, decisions, etc. necessary for controlling the assigned function.

[0035] The following explanation will describe each component with reference to Figures 2 to 4.

[0036] First, with reference to Figure 2, the image acquisition unit 100 according to an embodiment of the present invention will be described. Referring to Figure 2, the image acquisition unit 100 can collect at least one image information and can divide at least one image information into a plurality of snippets. For example, the image acquisition unit 100 can collect at least one image information captured by an external imaging device equipped to capture images in real time as input information. The image acquisition unit 100 also divides at least one image information into a plurality of snippets, and divides at least one image information into a predetermined number of snippets (for example, T snippets) SN1, SN2, ..., SN T It can be divided into parts.

[0037] Furthermore, the image acquisition unit 100 can also divide at least one image information into multiple snippets and group and manage snippets that correspond to each other among at least one image information. However, this is merely illustrative and is not necessarily limited to this. For example, the image acquisition unit 100 may be configured to acquire only image information captured by an external imaging device, and the function of dividing the image information into multiple snippets may be performed by a separate device from the image acquisition unit 100. The image acquisition unit 100 can transmit information about the divided multiple snippets to the information extraction unit 200.

[0038] Next, with reference to Figure 3, an information extraction unit 200 according to an embodiment of the present invention will be described. Referring to Figure 3, the information extraction unit 200 receives information about a plurality of divided snippets and, based on this, can extract visual feature information and text feature information for each of the plurality of snippets. For this purpose, the information extraction unit 200 can include a visual feature extraction unit 210 and a text feature extraction unit 220. The visual feature extraction unit 210 can include a preprocessing unit 211 and a multi-scale correction unit 212, and the text feature extraction unit 220 can include a caption generation unit 221, a text embedding unit 222, and a multi-scale correction unit 223. Figure 3 mainly shows the components according to an embodiment of the present invention, and it goes without saying that when actually realizing the information extraction unit 200, fewer or more components may be included.

[0039] The following describes each component. The visual feature extraction unit 210 can extract visual feature information contained in each of the multiple snippets based on the information about the multiple snippets provided by the image acquisition unit 100. For example, the visual feature extraction unit 210 can extract visual feature information contained in each of the multiple snippets via a pre-trained neural network, ResNet-50, and the visual feature information may be information using the I3D (Inflated 3D ConvNet) method. However, this is merely an example and is not necessarily limited to this.

[0040] Furthermore, the visual feature extraction unit 210 can perform a preprocessing process via the preprocessing unit 211 to improve the consistency of the information extracted during visual feature information extraction. For example, the preprocessing unit 211 can perform preprocessing on each of the divided snippets, cropping the frame according to a predetermined number of cropping steps. For example, if the predetermined number of cropping steps is 5, the preprocessing unit 211 can crop each of the frames of the multiple snippets into the four corners and the center.

[0041] The visual feature extraction unit 210 can extract visual feature information for each of the pre-processed snippets after the multiple snippets have been pre-processed by the pre-processing unit 211. By performing pre-processing on each of the multiple snippets, which involves cropping the frame, the amount of information that can be processed increases compared to the existing information on multiple snippets. This allows for improved information consistency by extracting visual feature information.

[0042] The text feature extraction unit 220 can extract text feature information contained in each of the multiple snippets based on the information about the multiple snippets provided by the image acquisition unit 100. For example, the text feature extraction unit 220 can generate captions for each of the multiple snippets via the caption generation unit 221. In this case, the caption generation unit 221 can also generate captions for consecutive frames using a sliding window strategy for each of the multiple snippets. Furthermore, the captions describe the events contained in each of the multiple snippets, and may describe, for example, the actions of people and the arrangement of objects contained in each of the multiple snippets. However, this is merely an example and is not necessarily limited to this.

[0043] The caption generation unit 221 can transmit information about the generated captions to the text embedding unit 222, which can then perform text embedding based on the generated captions. For example, the text embedding unit 222 can perform text embedding using a drop-out noise method or based on pairs recorded in a natural language inference dataset, which is one of the natural language processing (NLP) methods. However, this is merely an example and is not necessarily limited to this. The text feature extraction unit 220 can extract text feature information by generating captions for each of the multiple snippets via the caption generation unit 221 and the text embedding unit 222 and performing text embedding.

[0044] On the other hand, the visual feature extraction unit 210 and the text feature extraction unit 220 can perform multi-scale correction on the visual feature information and text feature information that have been first extracted for each of the multiple snippets before transmitting the visual feature information and text feature information to the outside. For this purpose, the visual feature extraction unit 210 and the text feature extraction unit 220 can be equipped with multi-scale correction units 212 and 223, respectively.

[0045] For example, the multi-scale correction units 212 and 223 can correct the visual feature information and text feature information extracted firsthand via a temporal network according to multiple scales. The temporal network can include a multiple-layer pyramid dilated convolution region and a non-local block (NLB). The pyramid dilated convolution for a time range can be used to learn representations for multiple snippets at multiple scales, and the non-local block can be used to learn the general temporal dependencies between multiple snippets. By performing multi-scale correction via the temporal network on the visual feature information and text feature information, respectively, in the multi-scale correction units 212 and 223, temporal dependencies can be better captured when the visual feature information and text feature information corrected and extracted by the visual feature extraction unit 210 and the text feature extraction unit 220 are subsequently synthesized.

[0046] As described above, the information extraction unit 200 can extract visual feature information via the visual feature extraction unit 210 and text feature information via the text feature extraction unit 220, and can provide the extracted visual feature information and text feature information to the sensing control unit 300.

[0047] Next, with reference to Figure 4, a sensing control unit 300 according to an embodiment of the present invention will be described. Referring to Figure 4, the sensing control unit 300 according to an embodiment of the present invention generates composite information by combining the visual feature information and text feature information extracted by the information extraction unit 200, and can detect abnormal signs of at least one image information based on the generated composite information. For this purpose, the sensing control unit 300 can include a synthesis unit 310, a judgment unit 320, and an abnormality detection unit 330. Figure 4 mainly shows the components according to an embodiment of the present invention, and in actual implementation of the sensing control unit 300, fewer or more components may be included.

[0048] The following describes each component. The synthesis unit 310 generates synthesized information by synthesizing the visual feature information and text feature information extracted and provided by the information extraction unit 200, and can generate synthesized information using at least one of several synthesis methods. These synthesis methods may include concatenation, addition, and product, but this is merely illustrative and not limited to these. Furthermore, when synthesized information is generated via addition and product, the synthesis unit 310 can add a fully concatenated layer to reduce the dimension of the visual feature information to the same dimension as the dimension of the text feature information, and then generate synthesized information for the visual and text feature information. However, this is illustrative and not limited to these.

[0049] The determination unit 320 can determine the image loss value based on the composite information generated by the composite unit 310, and can determine a first loss value and a second loss value for determining the image loss value. For example, the first loss value may be a value corresponding to the total loss value of at least one image information collected by the image acquisition unit 100, and the second loss value may be a value corresponding to a learned value for detecting abnormal signs in at least one image information.

[0050] Specifically, the determination unit 320 can determine the feature size corresponding to each of the multiple snippets based on the composite information generated by the synthesis unit 310. In this case, the determination unit 320 can determine the feature size using the l2norm method, but this is merely an example and is not necessarily limited to this. Furthermore, the determination unit 320 can determine the average feature size of the feature sizes that correspond to a predetermined number of feature sizes in order of size from among the determined feature sizes. For example, if the predetermined number of feature sizes is k, the determination unit 320 can extract the top k feature sizes corresponding to each of the multiple snippets in order of size, and determine the average feature size based on these. Average feature size f FM (ν;k) can be determined by the following equation 1. In particular, the average feature size may be considered to be the feature size corresponding to at least one image information.

[0051]

number

[0052] Based on the average feature size determined via formula 1, the determination unit 320 uses formula 2 below to determine a first loss value L corresponding to the total loss value of at least one image information. fm It is possible to make that determination.

[0053]

number

[0054] 1st OL value L fm This can mean the total loss value, which includes both normal and abnormal data in at least one image.

[0055] Furthermore, the decision unit 320 can determine the feature size corresponding to each of the multiple snippets based on the generated composite information, and determine the frame level prediction value for each of the multiple snippets based on this. For example, the decision unit 320 can train a separately provided binary snippet classifier based on a predetermined number of feature sizes in order of size from the determined feature sizes. Then, based on the trained binary snippet classifier, the decision unit 320 can determine the snippet level prediction value propagated to the individual frames contained in each of the multiple snippets, and determine the frame level prediction value for each of the multiple snippets based on the determined snippet level prediction value.

[0056] Furthermore, the determination unit 320 can determine the number of anomalies in at least one image information based on the frame level prediction values ​​corresponding to a predetermined number of feature sizes in order of size among the feature sizes corresponding to each of the multiple snippets. The determination unit 320 assumes that it determines the number of anomalies for at least one entire image information based on the values ​​corresponding to the highest number of feature sizes in order of feature size among the frame level prediction values ​​corresponding to each of the multiple snippets. As a result, the determination unit 320 can determine the number of anomalies fs(ν;k) for at least one image information via the following formula 3.

[0057]

number

[0058] At this time, f pred(Xi;δ) can represent the frame level prediction value corresponding to a predetermined number of feature sizes in order of size, from among the feature sizes corresponding to each of the multiple snippets. An incidental objective of the present invention is to facilitate the distinction between normal and abnormal data by increasing the difference between the number of abnormal points in image information containing normal data and the number of abnormal points in image information containing abnormal data. To achieve this, a binary snippet classifier can be trained based on a predetermined number of feature sizes in order of size, from among the feature sizes corresponding to each of the multiple snippets. Then, by determining the number of abnormal points in the image information based on the frame level prediction value determined by the trained binary snippet classifier, it is possible to increase the difference between the number of abnormal points in image information containing normal data and the number of abnormal points in image information containing abnormal data.

[0059] Subsequently, the determination unit 320 can determine a second loss value corresponding to a learned value for detecting abnormal signs in at least one image information, based on the number of abnormal points fs(ν;k) for at least one image information determined via the formula 3. At this time, the determination unit 320 determines the second loss value L via the following formula 4, which is constructed based on binary cross entropy. bce It is possible to make that determination.

[0060]

number

[0061] Then, the determination unit 320 determines the first loss value L as described above. fm and the second loss value L bce A loss function can be generated based on this. In this case, the decision unit 320 can generate the loss function L using the following formula 5.

[0062]

number

[0063] In the formula, α is an intermediate variable for adjusting the weight of the loss component. In the embodiments of the present invention, the weight of the first loss value can be adjusted. That is, the loss function can be generated as a linear function based on the intermediate variable α, the first loss value L fm and the second loss value L bce However, this is merely exemplary and not necessarily limited thereto.

[0064] The determination unit 320 can provide the generated loss function to the abnormality detection unit 330, and the abnormality detection unit 330 can detect an abnormality sign of at least one piece of image information based on the generated loss function. When an abnormality sign is detected from at least one piece of image information, the abnormality detection unit 330 can output information about the detected abnormality sign so that an external user can recognize the abnormality sign.

[0065] For example, when an abnormality sign is detected from at least one piece of image information, the abnormality detection unit 330 can output the abnormality score of at least one piece of image information and the caption included in at least one piece of image information. At this time, the abnormality score of at least one piece of image information may be an abnormality score for a plurality of snippets obtained by dividing at least one piece of image information, and the caption included in at least one piece of image information may be a caption for a plurality of snippets obtained by dividing at least one piece of image information.

[0066] Furthermore, when the anomaly detection unit 330 attempts to output the number of anomalies in at least one image information and the caption included in at least one image information, it can also output a warning alarm. For example, when the anomaly detection unit 330 detects an anomaly, it can compare the determined number of anomalies in at least one image information with a predetermined reference score. If, as a result of the comparison described above, the number of anomalies in at least one image information exceeds the predetermined reference score, the anomaly detection unit 330 can generate a warning alarm. By having the anomaly detection unit 330 generate a warning alarm and outputting the generated warning alarm, external users can recognize the anomaly and respond accordingly. However, this is merely an example and is not necessarily limited to this.

[0067] Hereinafter, a method for detecting image abnormalities according to an embodiment of the present invention will be described with reference to Figure 5, based on the configuration of the device for detecting image abnormalities described above with reference to Figures 1 to 4. However, a detailed explanation of each step in Figure 5 has been described above with reference to Figures 1 to 4, so it will be omitted below.

[0068] Figure 5 is a flowchart illustrating a method for detecting image anomalies according to one embodiment of the present invention. Referring to Figure 5, the image acquisition unit 100 can collect at least one image from an external imaging device (S510), and can divide the collected image information into multiple snippets (S520). The information extraction unit 200 can then extract visual feature information and text feature information for each of the multiple snippets based on the divided snippets (S530).

[0069] Furthermore, the sensing control unit 300 can synthesize the visual feature information and text feature information extracted by the information extraction unit 200 to generate synthesized information (S540), and based on the generated synthesized information, it can determine a first loss value corresponding to the total loss value of at least one image information, and a second loss value corresponding to a learned value for detecting anomaly signs in at least one image information (S550). Based on the determined first loss value and second loss value, the sensing control unit 300 can generate a loss function (S560), and based on the generated loss function, it can perform anomaly detection for at least one image information (S570).

[0070] Subsequently, when an anomaly detection is performed, if an anomaly is detected from at least one image information (Yes in S580), the detection control unit 300 can output the number of anomalies in at least one image information and the caption included in at least one image information (S590). Furthermore, if the number of anomalies in at least one image information exceeds a predetermined reference number, the detection control unit 300 can also output a warning alarm at the time of outputting both the number of anomalies and the caption.

[0071] As described above, the method and apparatus for detecting image anomalies of the present invention can improve the accuracy of detecting image anomalies by detecting anomaly signs contained in an image based on visual and text features extracted from the image. Furthermore, the robustness of the method and apparatus for detecting image anomalies can be improved by performing multi-scale correction on both the visual and text features. In addition, by outputting the number of anomaly points in the image and the caption contained in the image, users of the method and apparatus for detecting image anomalies according to the present invention can efficiently respond to anomaly signs contained in the image.

[0072] Furthermore, since the semantic meaning of abnormal signs contained in an image can be captured by using not only visual features but also textual features, it can be applied to a wide range of fields. Although specific embodiments of the present invention have been illustrated and described, it will be obvious to those ordinary skill in the art that various improvements and modifications can be made to the present invention without departing from the technical spirit of the invention provided by the following claims.

[0073] The present invention described above can be implemented as computer-readable code on a medium on which a program is recorded. Computer-readable media include all types of recording devices that store data to be read by a computer system. Examples of computer-readable media include HDDs (Hard-Disk Drives), SSDs (Solid State Disks), SDDs (Silicon Disk Drives), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and the like. Therefore, the above detailed description should not be interpreted restrictively in any way, but should be considered illustrative. The scope of the present invention must be determined by a reasonable analysis of the appended claims, and all modifications within the equivalent scope of the present invention are included within the scope of the present invention. [Explanation of symbols]

[0074] 100 Image Collection Department 200 Information extraction section 210 Visual Feature Extraction Unit 220 Text Feature Extraction Unit 300 Sensing Control Unit

Claims

1. The steps include: collecting at least one image information and dividing the at least one image information into multiple snippets; The steps include extracting visual feature information and text feature information for each of the divided snippets, The steps include generating composite information by combining the extracted visual feature information and text feature information, A method for detecting image anomalies, comprising the step of detecting an abnormality in at least one image information based on the generated composite information.

2. The extraction step described above is: The steps include: performing a preprocessing operation on each of the divided snippets, in which the frame is cropped according to a predetermined number of cropping cycles; The method for detecting image anomalies according to claim 1, comprising the step of extracting the visual feature information for each of the multiple snippets that have undergone the aforementioned preprocessing.

3. The extraction step described above is: The steps include generating captions for each of the divided snippets, A method for detecting an image anomaly according to claim 1, comprising the step of extracting the text feature information by performing text embedding based on the generated caption.

4. The extraction step described above is: The steps include applying multi-scale correction to the visual feature information and text feature information for each of the plurality of snippets, The method for detecting image anomalies according to claim 1, characterized by comprising the step of extracting the corrected visual feature information and text feature information.

5. The above generation step is, The method for detecting image anomalies according to claim 1, characterized in that it includes the step of generating the synthesized information using at least one of a plurality of synthesis methods from the extracted visual feature information and text feature information.

6. The aforementioned sensing step is, Based on the generated composite information, the steps include determining a first loss value corresponding to the total loss value of the at least one image information, and a second loss value corresponding to a learned value for detecting abnormal signs in the at least one image information, The steps include generating a loss function based on the first and second loss values ​​determined above, The method for detecting an image anomaly according to claim 1, comprising the step of detecting an anomaly in at least one image information based on the generated loss function.

7. The steps for determining the first loss value and the second loss value are as follows: The steps include determining the feature size corresponding to each of the multiple snippets based on the generated composite information, The steps include determining the average feature size of the feature sizes that correspond to a predetermined standard number in order of size from the feature sizes determined above, A method for detecting an image anomaly according to claim 6, comprising the step of determining the first loss value based on the average feature size determined above.

8. The steps for determining the first loss value and the second loss value are as follows: The steps include determining the feature size corresponding to each of the multiple snippets based on the generated composite information, The steps include determining the frame level prediction value for each of the plurality of snippets based on the determined feature size, The steps include determining the number of anomalies in at least one image information based on frame level prediction values ​​corresponding to a predetermined number of feature sizes in order of size from the aforementioned feature sizes, The method for detecting image anomalies according to claim 6, comprising the step of determining the second loss value based on the number of anomaly points determined above.

9. After the sensing step, The method for detecting image abnormalities according to claim 1, further comprising the step of outputting the number of abnormal points in the at least one image information and the caption included in the at least one image information when the abnormality signs are detected.

10. The output step described above is: The method for detecting image anomalies according to claim 9, characterized in that it includes the step of outputting a warning alarm when the number of anomaly points in at least one image information exceeds a predetermined reference number.

11. An image acquisition unit that collects at least one image information and divides the at least one image information into multiple snippets, An information extraction unit extracts visual feature information and text feature information for each of the multiple snippets divided by the image acquisition unit, An image anomaly detection device comprising: a sensing control unit that synthesizes the extracted visual feature information and text feature information to generate synthesized information, and senses an abnormality in at least one image piece based on the generated synthesized information.

12. The information extraction unit, The device for detecting image anomalies according to claim 11, characterized in that each of the divided snippets is subjected to preprocessing in which the frame is cropped according to a predetermined number of cropping cycles, and the visual feature information is extracted for each of the preprocessed snippets.

13. The information extraction unit, The image anomaly detection device according to claim 11, characterized by generating captions for each of the divided snippets, and extracting the text feature information by performing text embedding based on the generated captions.

14. The information extraction unit, The image anomaly detection device according to claim 11, characterized in that it applies multi-scale correction to the visual feature information and text feature information for each of the plurality of snippets, and extracts the corrected visual feature information and text feature information.

15. The sensing control unit, The image anomaly detection device according to claim 11, characterized in that the extracted visual feature information and text feature information are combined using at least one of a plurality of combination methods to generate the combined information.

16. The sensing control unit, The image anomaly detection device according to claim 11, characterized in that it determines a first loss value corresponding to the total loss value of the at least one image information and a second loss value corresponding to a learned value for detecting an anomaly in the at least one image information based on the generated composite information, generates a loss function based on the determined first loss value and second loss value, and detects an anomaly in the at least one image information based on the generated loss function.

17. The sensing control unit, The image anomaly detection device according to 16, characterized in that it determines the feature size corresponding to each of the plurality of snippets based on the generated composite information, determines the average feature size of a predetermined number of feature sizes in order of size from among the determined feature sizes, and determines the first loss value based on the determined average feature size.

18. The sensing control unit, The image anomaly detection device according to 16, characterized in that it determines the feature size corresponding to each of the plurality of snippets based on the generated composite information, determines the frame level prediction value for each of the plurality of snippets based on the determined feature size, determines the number of anomalies in at least one image information based on the frame level prediction values ​​corresponding to a predetermined number of feature sizes in order of size, and determines the second loss value based on the determined number of anomalies.

19. The sensing control unit, The image anomaly detection device according to claim 11, characterized in that when the aforementioned anomaly signs are detected, the number of anomaly points in at least one image information and the caption included in the at least one image information are output.

20. The sensing control unit, The image anomaly detection device according to claim 19, characterized in that a warning alarm is output when the number of anomaly points in at least one image information exceeds a predetermined reference number.

Images