Method and apparatus for detecting image abnormality

Abstract

The present invention relates to a method for detecting an image abnormality and an apparatus for detecting an image abnormality. The method for detecting an image abnormality according to an embodiment may comprise the steps of: extracting a text feature and a visual feature from an input image divided into a plurality of snippets; synthesizing the extracted features; and detecting an image abnormality on the basis of the synthesis result.

Description

Method and device for detecting image abnormalities

[0001] The present invention relates to a method and device for detecting an image abnormality capable of detecting an abnormality sign included in an image.

[0002]

[0003] Recently, video anomaly detection, which identifies abnormal data in video without direct human observation, has been applied in various fields. For example, in manufacturing, it can detect abnormal behavior and irregular operations in production processes. In the medical field, it can monitor patient conditions and automatically send alarms to medical staff in the event of an accident. Furthermore, in public safety, it aims to protect lives and property by preemptively recognizing illegal or accidental activities in video.

[0004] For this type of image anomaly detection to work smoothly, a balanced mix of normal and abnormal data is required. However, in everyday life, events corresponding to abnormal data are rare, resulting in a shortage of abnormal data. This creates an imbalance between normal and abnormal data, making it difficult to perform smooth image anomaly detection. Furthermore, the unclear criteria for distinguishing between normal and abnormal data also hinders smooth image anomaly detection.

[0005] To address these issues, video anomaly detection was traditionally performed based on information about visual features contained in the video, depending on the degree of supervision by the video observer. However, based solely on visual features, it is difficult to clearly determine the semantic meaning contained in the video, frequently resulting in a decrease in the accuracy of video anomaly detection.

[0006]

[0007] The matters described as background technology above are only intended to enhance understanding of the background of the present invention, and should not be taken as an admission that they correspond to prior art already known to those skilled in the art.

[0008]

[0009] The present invention has been proposed to solve such problems, and aims to provide a method and device for detecting image abnormalities by using not only visual features but also text features included in the image to detect abnormal signs included in the image.

[0010]

[0011] The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0012]

[0013] A method for detecting an image abnormality according to the present invention for achieving the above object may include the steps of collecting at least one image information and dividing the at least one image information into a plurality of snippets; extracting visual feature information and text feature information for each of the plurality of divided snippets; generating composite information by synthesizing the extracted visual feature information and text feature information; and detecting an abnormality sign of the at least one image information based on the generated composite information.

[0014] For example, the extracting step may include a step of performing preprocessing in which frames are cropped according to a preset number of crops for each of the plurality of segmented snippets; and a step of extracting the visual feature information for each of the plurality of snippets on which the preprocessing has been performed.

[0015] For example, the extracting step may include a step of generating a caption included in each of the plurality of segmented snippets; and a step of extracting the text feature information by performing sentence embedding based on the generated caption.

[0016] For example, the extracting step may include a step of performing multi-scale correction on the visual feature information and the text feature information for each of the plurality of snippets; and a step of extracting the visual feature information and the text feature information on which the correction has been performed.

[0017] For example, the generating step may include a step of generating the synthetic information using at least one of a plurality of synthetic methods for the extracted visual feature information and text feature information.

[0018] For example, the detecting step may include: a step of determining a first loss value corresponding to a total loss value of the at least one piece of image information and a second loss value corresponding to a learning value for detecting an abnormality sign of the at least one piece of image information based on the generated synthetic information; a step of generating a loss function based on the determined first loss value and second loss value; and a step of detecting an abnormality sign of the at least one piece of image information based on the generated loss function.

[0019] For example, the step of determining the first loss value and the second loss value, respectively, may include: a step of determining a feature size corresponding to each of the plurality of snippets based on the generated synthetic information; a step of determining an average feature size of feature sizes corresponding to a preset reference number in size order among the determined feature sizes; and a step of determining the first loss value based on the determined average feature size.

[0020] For example, the step of determining the first loss value and the second loss value, respectively, may include: a step of determining a feature size corresponding to each of the plurality of snippets based on the generated synthetic information; a step of determining a frame-level prediction value of each of the plurality of snippets based on the determined feature size; a step of determining an abnormality score of the at least one piece of image information based on a frame-level prediction value corresponding to a feature size corresponding to a preset reference number in order of increasing feature size; and a step of determining the second loss value based on the determined abnormality score.

[0021] For example, after the detecting step, if the abnormality is detected, the method may further include a step of outputting an abnormality score of the at least one piece of image information and a caption included in the at least one piece of image information.

[0022] For example, the outputting step may include a step of outputting a warning alarm when the abnormality score of at least one piece of image information exceeds a preset reference score.

[0023]

[0024] In addition, a device for detecting an image abnormality according to the present invention for achieving the above purpose may include an image collection unit that collects at least one image information and divides the at least one image information into a plurality of snippets; an information extraction unit that extracts visual feature information and text feature information for each of the plurality of snippets divided by the image collection unit; and a detection control unit that synthesizes the extracted visual feature information and text feature information to generate synthesized information, and detects an abnormality sign of the at least one image information based on the generated synthesized information.

[0025] For example, the information extraction unit may perform preprocessing in which frames are cropped according to a preset number of crops for each of the plurality of segmented snippets, and extract the visual feature information for each of the plurality of snippets for which the preprocessing has been performed.

[0026] For example, the information extraction unit can generate a caption included in each of the plurality of segmented snippets and perform sentence embedding based on the generated caption to extract the text feature information.

[0027] For example, the information extraction unit may perform multi-scale correction on the visual feature information and the text feature information for each of the plurality of snippets, and extract the visual feature information and text feature information on which the correction has been performed.

[0028] For example, the detection control unit can generate the synthetic information by using at least one of a plurality of synthetic methods for the extracted visual feature information and text feature information.

[0029] For example, the detection control unit may determine a first loss value corresponding to a total loss value of the at least one piece of image information and a second loss value corresponding to a learning value for detecting an abnormality sign of the at least one piece of image information based on the generated synthetic information, generate a loss function based on the determined first loss value and second loss value, and detect an abnormality sign of the at least one piece of image information based on the generated loss function.

[0030] For example, the detection control unit may determine a feature size corresponding to each of the plurality of snippets based on the generated synthetic information, determine an average feature size of feature sizes corresponding to a preset reference number in size order among the determined feature sizes, and determine the first loss value based on the determined average feature size.

[0031] For example, the detection control unit may determine a feature size corresponding to each of the plurality of snippets based on the generated synthetic information, determine a frame-level prediction value of each of the plurality of snippets based on the determined feature size, determine an abnormality score of the at least one image information based on a frame-level prediction value corresponding to a feature size corresponding to a preset reference number in order of increasing feature size, and determine the second loss value based on the determined abnormality score.

[0032] For example, the detection control unit may cause the abnormality score of the at least one piece of image information and the caption included in the at least one piece of image information to be output when the abnormality sign is detected.

[0033] For example, the detection control unit may output a warning alarm when the abnormality score of at least one piece of image information exceeds a preset reference score.

[0034]

[0035] As described above, the method and device for detecting image abnormalities of the present invention can improve the accuracy in detecting image abnormalities by detecting abnormal signs included in an image based on visual features and text features extracted from the image.

[0036] In addition, the robustness of a method and device for detecting image anomalies can be improved by performing multi-scale correction for each of visual features and text features.

[0037] In addition, by outputting an abnormality score for an image and a caption included in the image, a user using the method and device for detecting an image abnormality according to the present invention can efficiently respond to an abnormality sign included in the image.

[0038] In addition, it can be applied to various fields because it can capture the semantic meaning of abnormal signs contained in images by using text features as well as visual features.

[0039]

[0040] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below.

[0041]

[0042] Figures 1 to 4 are block diagrams for explaining the configuration of a device for detecting image abnormalities according to one embodiment of the present invention.

[0043] FIG. 5 is a flowchart illustrating a method for detecting image abnormalities according to one embodiment of the present invention.

[0044]

[0045] In describing the embodiments disclosed in this specification, detailed descriptions of related known technologies will be omitted if it is determined that such detailed descriptions may obscure the gist of the embodiments disclosed in this specification. In addition, the attached drawings are provided solely to facilitate understanding of the embodiments disclosed in this specification, and the technical concepts disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included within the spirit and technical scope of the present invention.

[0046] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0047] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0048] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0049] In this specification, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0050] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components are given the same reference numbers and redundant descriptions thereof will be omitted.

[0051] Referring to FIGS. 1 to 4, a device for detecting image abnormalities according to an embodiment of the present invention will be described.

[0052]

[0053] Figures 1 to 4 are block diagrams for explaining the configuration of a device for detecting image abnormalities according to one embodiment of the present invention.

[0054] Referring to Fig. 1, a device for detecting an image abnormality according to an embodiment of the present invention may include an image collection unit (100), an information extraction unit (200), and a detection control unit (300). Fig. 1 mainly illustrates components related to an embodiment of the present invention, and it is obvious that when implementing an actual device for detecting an image abnormality, fewer or more components may be included.

[0055] Meanwhile, each component (100, 200, 300) of the device for detecting image abnormalities according to an embodiment of the present invention may include a communication device for communicating with other components, a memory for storing operating systems or logic commands and input / output information, and one or more processors for performing judgments, calculations, decisions, etc. necessary for controlling the responsible function.

[0056] Below, each component will be described with reference to Figures 2 to 4.

[0057] First, an image collection unit (100) according to an embodiment of the present invention will be described with reference to FIG. 2.

[0058] Referring to FIG. 2, the image collection unit (100) can collect at least one image information and divide the at least one image information into a plurality of snippets. For example, the image collection unit (100) can collect at least one image information captured by an external shooting device equipped to capture an image in real time as input information. In addition, the image collection unit (100) divides the at least one image information into a plurality of snippets, and divides the at least one image information into a plurality of snippets (SN1, SN2, SN3) by a preset number (e.g., T). , SN T ) can be divided into.

[0059] In addition, the image collection unit (100) may divide at least one image information into multiple snippets, and group and manage corresponding snippets among the at least one image information. However, this is merely exemplary and the present invention is not necessarily limited thereto. For example, the image collection unit (100) may be configured to collect only image information captured by an external photographing device, and accordingly, the function of dividing the image information into multiple snippets may be performed separately from the image collection unit (100) by a separate device.

[0060] The image collection unit (100) can transmit information about a plurality of segmented snippets to the information extraction unit (200).

[0061] Next, the information extraction unit (200) according to an embodiment of the present invention will be described with reference to FIG. 3.

[0062]

[0063] Referring to FIG. 3, the information extraction unit (200) may be provided with information on a plurality of segmented snippets, and based on the information, may extract visual feature information and text feature information for each of the plurality of snippets. To this end, the information extraction unit (200) may include a visual feature extraction unit (210) and a text feature extraction unit (220). In addition, the visual feature extraction unit (210) may include a preprocessing unit (211) and a multi-scale correction unit (212), and the text feature extraction unit (220) may include a caption generation unit (221), a sentence embedding unit (222), and a multi-scale correction unit (223). FIG. 3 mainly illustrates components related to an embodiment of the present invention, and it goes without saying that the actual information extraction unit (200) may include fewer or more components.

[0064] Below, each component is described.

[0065] The visual feature extraction unit (210) can extract visual feature information included in each of the plurality of snippets based on information about the plurality of snippets provided by the image collection unit (100). For example, the visual feature extraction unit (210) can extract visual feature information included in each of the plurality of snippets through ResNet-50, which is a pre-trained neural network, and the visual feature information may be information according to the I3D (Inflated 3D Convnet) method. However, this is merely an example and is not necessarily limited thereto.

[0066] In addition, the visual feature extraction unit (210) may perform a preprocessing process through the preprocessing unit (211) to increase the consistency of information when extracting visual feature information. For example, the preprocessing unit (211) may perform preprocessing in which the frames of each of the plurality of segmented snippets are cropped according to a preset cropping number. For example, when the preset cropping number is 5, the preprocessing unit (211) may crop the frames of each of the plurality of snippets to the four corners and the center.

[0067] The visual feature extraction unit (210) can extract visual feature information for each of the plurality of snippets on which preprocessing has been performed when the plurality of snippets have been preprocessed through the preprocessing unit (211). By performing preprocessing in which frames are cropped for each of the plurality of snippets, the amount of information that can be processed increases compared to the information for the existing plurality of snippets, and by extracting visual feature information through this, the consistency of information can be improved.

[0068] The text feature extraction unit (220) can extract text feature information included in each of the plurality of snippets based on information about the plurality of snippets provided by the image collection unit (100). For example, the text feature extraction unit (220) can generate captions included in each of the plurality of snippets through the caption generation unit (221). At this time, the caption generation unit (221) can also generate captions for consecutive frames using a sliding window strategy for each of the plurality of snippets. In addition, the captions describe events included in each of the plurality of snippets, for example, they can describe the movements of people and the arrangement of objects included in each of the plurality of snippets. However, this is merely an example and the present invention is not limited thereto.

[0069] The caption generation unit (221) can transmit information about the generated caption to the sentence embedding unit (222), and the sentence embedding unit (222) can perform sentence embedding based on the generated caption. For example, the sentence embedding unit (222) can perform sentence embedding using a drop-out noise method or based on pairs recorded in a natural language inference dataset, which is one of the natural language processing (NLP) methods. However, this is merely an example, and the present invention is not necessarily limited thereto. The text feature extraction unit (220) can extract text feature information by generating captions for each of a plurality of snippets through the caption generation unit (221) and the sentence embedding unit (222), and performing sentence embedding.

[0070] Meanwhile, the visual feature extraction unit (210) and the text feature extraction unit (220) may perform multi-scale correction on the initially extracted visual feature information and text feature information for each of the plurality of snippets before transmitting the visual feature information and text feature information to the outside. To this end, the visual feature extraction unit (210) and the text feature extraction unit (220) may each be equipped with multi-scale correction units (212, 223).

[0071] For example, the multi-scale correction unit (212, 223) can perform multi-scale correction on the visual feature information and text feature information that are initially extracted through a temporal network. The temporal network can include a multiple layer pyramid dilated convolution region and a non-local region (NLB: Non-Local Block). The pyramid dilated convolution for the temporal range can be used to learn representations for multiple snippets at multiple scales, and the non-local region can be used to learn the overall temporal dependency between the multiple snippets. By performing multi-scale correction on the visual feature information and the text feature information through the temporal network in the multi-scale correction unit (212, 223), the temporal dependency can be well captured when the visual feature information and the text feature information that are corrected and extracted in the visual feature extraction unit (210) and the text feature extraction unit (220) are synthesized.

[0072] As described above, the information extraction unit (200) can extract visual feature information through the visual feature extraction unit (210), extract text feature information through the text feature extraction unit (220), and provide the extracted visual feature information and text feature information to the detection control unit (300).

[0073] Next, a detection control unit (300) according to an embodiment of the present invention will be described with reference to FIG. 4.

[0074]

[0075] Referring to FIG. 4, the detection control unit (300) according to an embodiment of the present invention can generate composite information by synthesizing the visual feature information and text feature information extracted from the information extraction unit (200), and detect an abnormality sign of at least one image information based on the generated composite information. To this end, the detection control unit (300) may include a synthesis unit (310), a judgment unit (320), and an abnormality detection unit (330). FIG. 4 mainly illustrates components related to an embodiment of the present invention, and it goes without saying that the actual detection control unit (300) may include fewer or more components.

[0076] Below, each component is described.

[0077] The synthesis unit (310) synthesizes the visual feature information and text feature information extracted and provided by the information extraction unit (200) to generate synthetic information, and may generate the synthetic information using at least one synthesis method among multiple synthesis methods. The multiple synthesis methods may include concatenation, addition, and multiplication methods, but these are merely examples and are not necessarily limited thereto.

[0078] In addition, when synthetic information is generated through the addition and multiplication method, the synthesis unit (310) can add a fully connected layer to reduce the dimension of the visual feature information to the same dimension as the text feature information, and then generate synthetic information for the visual feature information and the text feature information. However, this is merely exemplary and should not be construed as being limited thereto.

[0079] The judgment unit (320) can determine an image loss value based on the synthesis information generated by the synthesis unit (310), and can determine a first loss value and a second loss value for determining the image loss value. For example, the first loss value may be a value corresponding to the total loss value of at least one piece of image information collected by the image collection unit (100), and the second loss value may be a value corresponding to a learning value for detecting an abnormality sign of at least one piece of image information.

[0080] Specifically, the judgment unit (320) can determine the feature size corresponding to each of the plurality of snippets based on the synthesis information generated by the synthesis unit (310). At this time, the judgment unit (320) can determine the feature size using the l2norm method, but this is merely exemplary and is not necessarily limited thereto.

[0081] And, the judgment unit (320) can determine the average feature size of the feature sizes corresponding to a preset standard number in order of size among the judged feature sizes. For example, if the preset standard number is k, the judgment unit (320) can extract the feature sizes corresponding to the top k in order of size among the feature sizes corresponding to each of a plurality of snippets, and determine the average feature size based on this. Average feature size ( ) can be determined through the following mathematical expression 1. In particular, the average feature size can be determined as the feature size corresponding to at least one piece of image information.

[0082]

[0083] The judgment unit (320) uses the mathematical expression 2 below based on the average feature size determined through the mathematical expression 1 above to determine the first loss value corresponding to the total loss value of at least one image information. ) can be judged.

[0084]

[0085] The first loss value ( ) may mean a total loss value that includes both normal data and abnormal data in at least one image information.

[0086] In addition, the judgment unit (320) can determine the feature size corresponding to each of the plurality of snippets based on the generated synthetic information, and can determine the frame-level prediction value of each of the plurality of snippets based on this. For example, the judgment unit (320) can train a separately provided binary snippet classifier based on feature sizes corresponding to a preset reference number in size order among the determined feature sizes. In addition, the judgment unit (320) can determine the snippet-level prediction value propagated to the individual frame included in each of the plurality of snippets based on the trained binary snippet classifier, and can determine the frame-level prediction value of each of the plurality of snippets based on the determined snippet-level prediction value.

[0087] And, the judgment unit (320) can judge the abnormality score of at least one piece of image information based on the frame-level prediction value corresponding to the feature size corresponding to the preset reference number in order of the feature size among the feature sizes corresponding to each of the plurality of snippets. It is assumed that the judgment unit (320) judges the abnormality score for at least one piece of image information as a whole based on the value corresponding to the upper reference number in order of the feature size among the frame-level prediction values ​​corresponding to each of the plurality of snippets. Accordingly, the judgment unit (320) can determine the abnormality score ( of at least one piece of image information) through the following mathematical expression 3. ) can be judged.

[0088]

[0089] At this time, may mean a frame-level prediction value corresponding to a feature size corresponding to a preset reference number in order of size among the feature sizes corresponding to each of a plurality of snippets. The present invention has a secondary purpose of facilitating the distinction between normal data and abnormal data by increasing the difference between the abnormality score of image information including normal data and the abnormality score of image information including abnormal data. To achieve this, a binary snippet classifier may be trained based on a feature size corresponding to a preset reference number in order of size among the feature sizes corresponding to each of a plurality of snippets. In addition, by determining the abnormality score of image information based on the frame-level prediction value determined by the trained binary snippet classifier, the difference between the abnormality score of image information including normal data and the abnormality score of image information including abnormal data may be increased.

[0090] Afterwards, the judgment unit (320) determines an abnormality score for at least one image information judged through the above mathematical expression 3. ) can be used to determine the second loss value corresponding to the learning value for detecting an abnormality of at least one image information. At this time, the determination unit (320) determines the second loss value ( through the following mathematical expression 4 formed based on binary cross entropy. ) can be judged.

[0091]

[0092] And, the judgment unit (320) determines the first loss value as described above ( ) and the second loss value ( ) can be used to generate a loss function. At this time, the judgment unit (320) uses the following mathematical expression 5 to generate the loss function ( ) can be created.

[0093]

[0094] In the above mathematical formula 5 is a parameter for adjusting the weight of the loss component, and in the embodiment of the present invention, the weight of the first loss value can be adjusted. That is, the loss function is a parameter as in the above mathematical expression 5. , the first loss value ( ) and the second loss value ( ) can be generated in the form of a linear function based on the equation. However, this is an example and is not necessarily limited to this.

[0095] The judgment unit (320) can provide the generated loss function to the anomaly detection unit (330), and the anomaly detection unit (330) can detect an anomaly sign of at least one piece of image information based on the generated loss function. In addition, when an anomaly sign is detected in at least one piece of image information, the anomaly detection unit (330) can output information about the detected anomaly sign so that an external user can recognize the anomaly sign.

[0096] For example, when an abnormality is detected in at least one piece of image information, the abnormality detection unit (330) may output an abnormality score of at least one piece of image information and a caption included in the at least one piece of image information. In this case, the abnormality score of the at least one piece of image information may be an abnormality score for a plurality of snippets into which the at least one piece of image information is divided, and the caption included in the at least one piece of image information may be a caption for a plurality of snippets into which the at least one piece of image information is divided.

[0097] In addition, the abnormality detection unit (330) may output a warning alarm together when outputting the abnormality score of at least one piece of image information and the caption included in the at least one piece of image information described above. For example, the abnormality detection unit (330) may compare the abnormality score of at least one piece of image information determined when an abnormality sign is detected with a preset reference score. The abnormality detection unit (330) may generate a warning alarm if the abnormality score of at least one piece of image information exceeds the preset reference score as a result of the preceding comparison. The abnormality detection unit (330) may generate a warning alarm and output the generated warning alarm so that an external user can recognize and respond to the abnormal sign. However, this is an example and is not necessarily limited thereto.

[0098]

[0099] Hereinafter, a method for detecting an image abnormality according to an embodiment of the present invention will be described with reference to FIG. 5 based on the configuration of the device for detecting an image abnormality described above with reference to FIGS. 1 to 4. However, since a specific description of each step of FIG. 5 has been described above with reference to FIGS. 1 to 4, a detailed description thereof will be omitted below.

[0100] FIG. 5 is a flowchart illustrating a method for detecting image abnormalities according to one embodiment of the present invention.

[0101] Referring to FIG. 5, the image collection unit (100) can collect at least one image information from an external photographing device (S510) and divide the collected at least one image information into a plurality of snippets (S520).

[0102] And, the information extraction unit (200) can extract visual feature information and text feature information for each of the plurality of snippets based on the divided plurality of snippets (S530).

[0103] In addition, the detection control unit (300) can generate synthetic information by synthesizing the visual feature information and text feature information extracted from the information extraction unit (200) (S540), and based on the generated synthetic information, can determine a first loss value corresponding to the total loss value of at least one piece of image information and a second loss value corresponding to a learning value for detecting an abnormality sign of at least one piece of image information (S550).

[0104] The detection control unit (300) can generate a loss function based on the determined first loss value and second loss value (S560), and can perform anomaly detection for at least one piece of image information based on the generated loss function (S570).

[0105] Thereafter, when performing anomaly detection, if an abnormality sign is detected in at least one piece of image information (Yes in S580), the detection control unit (300) can cause the abnormality score of at least one piece of image information and the caption included in the at least one piece of image information to be output (S590). In addition, the detection control unit (300) can cause a warning alarm to be output together with the abnormality score and caption output when the abnormality score of at least one piece of image information exceeds a preset reference score.

[0106]

[0107] As described above, the method and device for detecting image abnormalities of the present invention can improve the accuracy in detecting image abnormalities by detecting abnormal signs included in an image based on visual features and text features extracted from the image.

[0108] In addition, the robustness of a method and device for detecting image anomalies can be improved by performing multi-scale correction for each of visual features and text features.

[0109] In addition, by outputting an abnormality score for an image and a caption included in the image, a user using the method and device for detecting an image abnormality according to the present invention can efficiently respond to an abnormality sign included in the image.

[0110] In addition, it can be applied to various fields because it can capture the semantic meaning of abnormal signs contained in images by utilizing not only visual features but also text features.

[0111]

[0112] While the present invention has been illustrated and described with respect to specific embodiments thereof, it will be apparent to those skilled in the art that the present invention may be variously improved and modified without departing from the technical spirit of the invention as defined by the claims below.

[0113] The present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. Computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. Therefore, the above detailed description should not be construed as limiting in any respect, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

[0114]

[0115] [Explanation of symbols]

[0116] 100: Video Collection Department

[0117] 200: Information Extraction Department

[0118] 300: Detection control unit

Claims

1. A step of collecting at least one image information and dividing the at least one image information into a plurality of snippets; A step of extracting visual feature information and text feature information for each of the above-mentioned divided plurality of snippets; A step of generating synthetic information by synthesizing the extracted visual feature information and text feature information; and A method for detecting an image abnormality, comprising: a step of detecting an abnormality sign of at least one image information based on the generated synthetic information; 2. In claim 1, The above extraction steps are A step of performing preprocessing in which frames are cropped according to a preset cropping number for each of the above-described multiple snippets; and A method for detecting an image abnormality, characterized by comprising a step of extracting visual feature information for each of a plurality of snippets on which the above preprocessing has been performed.

3. In claim 1, The above extraction steps are A step of generating a caption included in each of the above-described multiple snippets; and A method for detecting an image abnormality, characterized by including a step of extracting the text feature information by performing sentence embedding based on the generated caption.

4. In claim 1, The above extraction steps are A step of performing multi-scale correction on the visual feature information and the text feature information for each of the plurality of snippets; and A method for detecting an image abnormality, characterized by comprising a step of extracting visual feature information and text feature information for which the above correction has been performed.

5. In claim 1, The above generating steps are A method for detecting an image abnormality, characterized by comprising the step of generating the synthetic information by using at least one of a plurality of synthetic methods of the extracted visual feature information and text feature information.

6. In claim 1, The above detection steps are A step of determining a first loss value corresponding to a total loss value of the at least one image information and a second loss value corresponding to a learning value for detecting an abnormality sign of the at least one image information based on the generated synthetic information; A step of generating a loss function based on the first loss value and the second loss value determined above; and A method for detecting an image abnormality, characterized by including a step of detecting an abnormality sign of at least one image information based on the generated loss function.

7. In claim 6, The steps of determining the first loss value and the second loss value are as follows: A step of determining a feature size corresponding to each of the plurality of snippets based on the generated synthetic information; A step of determining the average feature size of feature sizes corresponding to a preset standard number in size order among the above-determined feature sizes; and A method for detecting an image abnormality, characterized by comprising a step of determining the first loss value based on the determined average feature size.

8. In claim 6, The steps of determining the first loss value and the second loss value are as follows: A step of determining a feature size corresponding to each of the plurality of snippets based on the generated synthetic information; A step of determining a frame-level prediction value of each of the plurality of snippets based on the determined feature size; A step of determining an abnormality score of at least one image information based on a frame-level prediction value corresponding to a feature size corresponding to a preset standard number in the order of the feature size increase; and A method for detecting an image abnormality, characterized by comprising a step of determining the second loss value based on the determined abnormality score.

9. In claim 1, After the above detection step, A method for detecting an image abnormality, characterized by further comprising the step of outputting an abnormality score of at least one image information and a caption included in the at least one image information when the above abnormality sign is detected.

10. In claim 9, The above output steps are A method for detecting an image abnormality, characterized by comprising a step of outputting a warning alarm when an abnormality score of at least one of the image information exceeds a preset reference score.

11. An image collection unit that collects at least one image information and divides the at least one image information into a plurality of snippets; An information extraction unit that extracts visual feature information and text feature information for each of a plurality of snippets segmented from the above image collection unit; and A device for detecting an image abnormality, comprising: a detection control unit for generating synthetic information by synthesizing the extracted visual feature information and text feature information, and detecting an abnormality sign of at least one image information based on the generated synthetic information.

12. In claim 11, The above information extraction section A device for detecting an image abnormality, characterized in that it performs preprocessing in which frames are cropped according to a preset number of crops for each of the plurality of segmented snippets, and extracts the visual feature information for each of the plurality of snippets on which the preprocessing has been performed.

13. In claim 11, The above information extraction section A device for detecting an image abnormality, characterized in that it generates a caption included in each of the plurality of segmented snippets and performs sentence embedding based on the generated caption to extract the text feature information.

14. In claim 11, The above information extraction section A device for detecting an image abnormality, characterized by performing multi-scale correction on the visual feature information and the text feature information for each of the plurality of snippets, and extracting the visual feature information and the text feature information on which the correction was performed.

15. In claim 11, The above detection control unit A device for detecting an image abnormality, characterized in that it generates the synthetic information by using at least one of a plurality of synthetic methods from the extracted visual feature information and text feature information.

16. In claim 11, The above detection control unit A device for detecting an image abnormality, characterized in that a first loss value corresponding to a total loss value of the at least one image information and a second loss value corresponding to a learning value for detecting an abnormality sign of the at least one image information are determined based on the generated synthetic information, a loss function is generated based on the determined first loss value and second loss value, and an abnormality sign of the at least one image information is detected based on the generated loss function.

17. In claim 16, The above detection control unit A device for detecting an image abnormality, characterized in that the device determines a feature size corresponding to each of the plurality of snippets based on the generated synthetic information, determines an average feature size of feature sizes corresponding to a preset standard number in size order among the determined feature sizes, and determines the first loss value based on the determined average feature size.

18. In claim 16, The above detection control unit A device for detecting an image abnormality, characterized in that the device determines a feature size corresponding to each of the plurality of snippets based on the generated synthetic information, determines a frame-level prediction value of each of the plurality of snippets based on the determined feature size, determines an abnormality score of at least one image information based on a frame-level prediction value corresponding to a feature size corresponding to a preset reference number in order of increasing feature size, and determines the second loss value based on the determined abnormality score.

19. In claim 11, The above detection control unit A device for detecting an image abnormality, characterized in that when the above abnormality sign is detected, an abnormality score of at least one image information and a caption included in the at least one image information are output.

20. In claim 19, The above detection control unit A device for detecting an image abnormality, characterized in that a warning alarm is output when the abnormality score of at least one of the image information exceeds a preset reference score.

Images