Image analysis device, image analysis method, and recording medium

Abstract

This image analysis device comprises: an image acquisition unit that acquires a processing target image; a related information acquisition unit that acquires processing target image-related information indicating at least a target object which appears in the processing target image; and an analysis unit that, on the basis of the processing target image-related information, determines identity between the processing target image and a reference image and / or evaluates the target object which appears in the processing target image.

Description

Image analysis device, image analysis method, and recording medium

[0001] The present disclosure relates to an image analysis device, an image analysis method, and a program.

[0002] A technology related to the present disclosure is disclosed in Patent Literature 1. Patent Literature 1 discloses a technology for searching for illegal content posted by unauthorized users who do not have the rights to the legitimate content. This technology detects, based on the content profile, content whose profile has identity with the profile of a detected illegal content as suspected illegal content. At least one of the thumbnail image, content length, content title, and user account name is used as the profile used as the criteria for determining identity.

[0003] Japanese Patent Application Laid-Open No. 2019-174929

[0004] There is a need for a technology that can accurately determine the identity of two images and evaluate objects that appear in the images. The technology disclosed in Patent Document 1 determines the identity of profiles, but does not determine the identity of images or evaluate objects that appear in the images.

[0005] One example of the objective of the present disclosure is to provide an image analysis device, an image analysis method, and a program that, in consideration of the above-mentioned problems, accurately determine the identity of two images and evaluate objects that appear in the images.

[0006] According to the present disclosure, there is provided an image analysis device having an image acquisition means for acquiring an image to be processed, a related information acquisition means for acquiring information related to the image to be processed that indicates at least an object appearing in the image to be processed, and an analysis means for performing at least one of determining the identity of the image to be processed and a reference image and evaluating the object appearing in the image to be processed based on the information related to the image to be processed.

[0007] Furthermore, according to the present disclosure, an image analysis method is provided in which one or more computers acquire an image to be processed, acquire information related to the image to be processed that indicates at least an object appearing in the image to be processed, and, based on the information related to the image to be processed, perform at least one of determining the identity of the image to be processed and a reference image, and evaluating the object appearing in the image to be processed.

[0008] Furthermore, according to the present disclosure, a program is provided that causes a computer to function as: an image acquisition means that acquires an image to be processed; a related information acquisition means that acquires information related to the image to be processed that indicates at least an object that appears in the image to be processed; and an analysis means that performs at least one of determining the identity of the image to be processed and a reference image, and evaluating the object that appears in the image to be processed, based on the information related to the image to be processed.

[0009] According to one aspect of the present disclosure, an image analysis device, an image analysis method, and a program are realized that accurately determine the identity of two images and accurately evaluate objects that appear in the images.

[0010] FIG. 1 is a diagram illustrating an example of a functional block diagram of an image analysis device according to the present disclosure; FIG. 2 is a flowchart illustrating an example of a processing flow of an image analysis device according to the present disclosure; FIG. 3 is a diagram illustrating an example of a hardware configuration of an image analysis device according to the present disclosure; and FIG. 4 is a diagram schematically illustrating an example of information processed by an image analysis device according to the present disclosure.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0012] First Embodiment Fig. 1 is a functional block diagram showing an overview of an image analysis device 10. Fig. 2 is a flowchart showing an example of the flow of processing executed by the image analysis device 10.

[0013] 1, the image analysis device 10 includes an image acquisition unit 11, a related information acquisition unit 12, and an analysis unit 13. These functional units execute the process shown in FIG.

[0014] First, the image acquisition unit 11 acquires an image to be processed (S10).

[0015] Next, the related information acquisition unit 12 acquires processing target image related information that indicates at least the object appearing in the processing target image acquired in S10 (S11).

[0016] Then, based on the processing target image related information acquired in S11, the analysis unit 13 performs at least one of determining whether the processing target image and the reference image are identical and evaluating the object appearing in the processing target image.

[0017] In this way, the image analysis device 10 can perform at least one of determining the identity of the image to be processed and the reference image, and evaluating the objects appearing in the image to be processed, based on the information related to the image to be processed that indicates at least the objects appearing in the image to be processed. The image analysis device 10 can accurately determine the identity of the two images and evaluate the objects appearing in the image.

[0018] Second Embodiment Overview An image analysis device 10 according to a second embodiment is a specific implementation of the configuration of the image analysis device 10 according to the first embodiment. This will be described in detail below.

[0019] "Hardware Configuration" First, an example of the hardware configuration of the image analysis device 10 will be described. Each functional unit of the image analysis device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. Software includes programs that are pre-loaded in the device before shipping, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0020] FIG. 3 is a block diagram illustrating an example of the hardware configuration of an image analysis device 10. As shown in FIG. 3, the image analysis device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The image analysis device 10 does not necessarily have to have the peripheral circuit 4A. Note that the image analysis device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

[0021] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a processing unit such as a CPU or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, touch panel, etc. Examples of output devices include a display, speaker, printer, mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0022] "Functional Configuration" Next, a detailed description will be given of the functional configuration of the image analysis device 10. Fig. 1 shows an example of a functional block diagram of the image analysis device 10. As shown in the figure, the image analysis device 10 has an image acquisition unit 11, a related information acquisition unit 12, and an analysis unit 13.

[0023] The image acquisition unit 11 acquires an image to be processed.

[0024] The "processing target image" is a moving image. The processing target image is an image that is the target of image analysis by the analysis unit 13, which will be described later. As will be described later, the analysis unit 13 performs at least one of "determining whether the processing target image and the reference image are identical" and "evaluating objects that appear in the processing target image." The image that is the target of such image analysis is input to the image analysis device 10 as the processing target image.

[0025] For example, a video that may have been published without the permission of the rights holder is the image to be processed. As an example, a video uploaded to a server on the Internet and published is the image to be processed. Videos uploaded to a server and published may include those published without the permission of the rights holder. By using the image uploaded to a server and published as the image to be processed and determining its identity with legitimate content, it is possible to detect videos published without the permission of the rights holder.

[0026] In another example, the target image is legitimate content, and the legitimate content can be the target image to evaluate objects that appear in the target image.

[0027] Note that the examples of images to be processed here are merely examples, and the present invention is not limited to these. Images other than the moving images exemplified here can also be used as the targets for identity determination and object evaluation by the analysis unit 13.

[0028] The related information acquisition unit 12 acquires information related to the image to be processed. The related information acquisition unit 12 can analyze the image to be processed and generate information related to the image to be processed. The related information acquisition unit 12 can generate information related to the image to be processed by using any technology such as object detection technology, person detection technology, face authentication technology, person matching technology, facial expression recognition technology, and voice recognition technology.

[0029] "Processing target image related information" is information related to the processing target image acquired by the image acquisition unit 11. Processing target image related information indicates at least the objects that appear in the processing target image. In addition to the objects that appear in the processing target image, processing target image related information can further indicate at least one of the following information: Appearance timing of each object Total appearance time of each object Appearance position in the image at each appearance timing of each object Action at each appearance timing of each object Emotion at each appearance timing of each object Speech content arranged in chronological order

[0030] This will be explained in detail below.

[0031] Information indicating objects that appear in the image to be processed (hereinafter referred to as "appearing object information") "Object" is a concept that includes at least one of people, other animals (hereinafter simply referred to as "animals"), and objects. Objects may be living creatures or objects that exist in the real world, or they may be fictional creatures or objects. Examples of fictional creatures or objects include, but are not limited to, characters and equipment (weapons, etc.) from anime, manga, etc. The fictional creatures or objects may be designed in two dimensions or three dimensions.

[0032] In one example, the target image-related information may indicate personal identification information of a person appearing in the target image. If multiple people appear in the target image, the target image-related information may indicate personal identification information for each of the multiple people.

[0033] "Personal identification information" is information that identifies multiple people from one another. The personal identification information may be a name. The personal identification information may also be face information (face image, facial features) or other information about physical appearance (physique, gait features, etc.). The personal identification information may also be information about clothing, belongings (accessories, etc.), etc. The personal identification information may also be information that combines some or all of the exemplified information.

[0034] For example, the related information acquisition unit 12 detects people from the processing target image and extracts the feature amount of the appearance of each detected person. Then, the related information acquisition unit 12 can use the extracted feature amount of the appearance of each person as at least a part of the processing target image related information.

[0035] The related information acquisition unit 12 can detect people and extract features for each frame image of the processing target image, which is a moving image. The related information acquisition unit 12 may process all frame images or only some of the frame images. For example, the related information acquisition unit 12 may process every predetermined number of frames.

[0036] Alternatively, the related information acquisition unit 12 may identify the name of each person based on the feature amount of the appearance of the person detected from the image to be processed. Then, the related information acquisition unit 12 may use the name of each person identified in this way as at least a part of the information related to the image to be processed.

[0037] For example, a person database may be generated in advance, in which the appearance features and names of each of a plurality of people are registered. In one example, a person database may be generated in which the appearance features and names of celebrities, such as actors, actresses, comedians, entertainers, athletes, and anime and manga characters, are registered. Examples of the appearance features of a person include, but are not limited to, facial features and other physical appearance features (physique, gait, etc.). These features may be two-dimensional features or three-dimensional features. The image analysis device 10 may include the person database, or an external device configured to be able to communicate with the image analysis device 10 may include the person database.

[0038] The related information acquisition unit 12 may then identify the name of a person appearing in the image to be processed by comparing the features registered in the person database with the features of the appearance of the person detected from the image to be processed.

[0039] In another example, the processing target image related information may indicate animal identification information of an animal appearing in the processing target image. When multiple animals appear in the processing target image, the processing target image related information may indicate animal identification information for each of the multiple animals. The animal identification information may indicate the species of dog, cat, horse, etc. The animal identification information may indicate detailed information such as color, breed of dog, breed of cat, breed of horse, etc. For example, the related information acquisition unit 12 detects various animals from the processing target image. Then, the related information acquisition unit 12 may use the species, breed, etc. of the detected animals as at least part of the processing target image related information. The related information acquisition unit 12 may generate the detailed information using any image analysis technology, a classifier generated by machine learning, etc.

[0040] The related information acquisition unit 12 can detect various animals for each frame image of the processing target image, which is a moving image. The related information acquisition unit 12 may process all frame images or only some of the frame images. For example, the related information acquisition unit 12 may process every predetermined number of frames.

[0041] In another example, the processing target image related information may indicate object identification information of an object appearing in the processing target image. When multiple objects appear in the processing target image, the processing target image related information may indicate object identification information for each of the multiple objects. The object identification information may indicate the type of object, such as a car, ship, airplane, refrigerator, or table. The object identification information may also indicate detailed information about the object, such as its model number, manufacturer, model, and color. For example, the related information acquisition unit 12 detects various objects from the processing target image. Then, the related information acquisition unit 12 may include the type of the detected object as at least a part of the processing target image related information. The related information acquisition unit 12 may generate the detailed information using any image analysis technology, a classifier generated by machine learning, or the like.

[0042] The related information acquisition unit 12 can detect various objects for each frame image of the processing target image, which is a moving image. The related information acquisition unit 12 may process all frame images or only some of the frame images. For example, the related information acquisition unit 12 may process every predetermined number of frames.

[0043] Information indicating the appearance timing of each object (hereinafter referred to as appearance timing information) The appearance timing information is indicated, for example, by the elapsed time from the beginning of the processing target image, which is a moving image, or by identification information of the frame image in which each object is detected. The related information acquisition unit 12 can generate appearance timing information based on the results of the processing to detect objects from the processing target image described above. The related information acquisition unit 12 can then use the generated appearance timing information as at least a part of the processing target image related information.

[0044] The appearance timing information may be indicated in other ways. For example, the image to be processed may be classified into a plurality of time periods for each unit time from the beginning. The appearance timing information may indicate whether each object appears in each time period. The unit time may be any value, such as a few seconds or a few minutes. For example, if the unit time is one minute, the plurality of time periods may be classified into a time period from the beginning until one minute has elapsed, a time period from one minute until two minutes has elapsed from the beginning, a time period from two minutes until three minutes has elapsed from the beginning, etc.

[0045] The related information acquisition unit 12 may determine that each object appears in each time period if it appears even for a moment in each time period, and generate appearance timing information indicating that. Alternatively, the related information acquisition unit 12 may determine that each object appears in each time period if it appears for a reference time or longer in each time period, and generate appearance timing information indicating that.

[0046] Information indicating the total appearance time of each object (hereinafter referred to as total appearance time information) The total appearance time information indicates the total time (length) of time that each object appears in the processing target image. The related information acquisition unit 12 can generate total appearance time information based on the results of the processing to detect objects from the processing target image described above. Then, the related information acquisition unit 12 can use the generated total appearance time information as at least a part of the processing target image related information.

[0047] When the above-mentioned appearance timing information indicates the elapsed time from the beginning or the identification information of the frame image, the related information acquisition unit 12 may calculate the total appearance time of each object as, for example, the product of the number of frame images in which each object appears and the time per frame image (= 1 / (frame rate)).

[0048] On the other hand, if the above-mentioned appearance timing information indicates the time period during which each object appears, the related information acquisition unit 12 may calculate, for example, the product of the number of time periods during which each object appears and the unit time as the total appearance time of each object.

[0049] Alternatively, the related information acquisition unit 12 may calculate the total appearance time of each object by counting the time from the time of appearance to the time of frame-out and measuring and adding up the time each time the object appears.

[0050] Information indicating the appearance position of each object in the image at each appearance timing (hereinafter referred to as appearance position information) The appearance position information is indicated, for example, by coordinates in a coordinate system set in the image. The information may indicate, for example, the position of a reference part of the object (e.g., a predetermined part of the face) in the image. Alternatively, if the area of ​​the object in the image is detected as a rectangular area, the information may indicate the position of this rectangular area. The position of the rectangular area may be indicated by the coordinates of its four vertices, by the coordinates of a predetermined vertex among the four vertices (e.g., the upper left vertex), by the coordinates of the intersection of two diagonals of the rectangular area, or by other means.

[0051] When the above-mentioned appearance timing information indicates the elapsed time from the beginning or the identification information of the frame image, the related information acquisition unit 12 can treat the appearance position of each object in the frame image in which each object appears as information indicating the appearance position.

[0052] On the other hand, when the appearance timing information indicates a time period in which each object appears, the related information acquisition unit 12 can use the appearance position of each object at a reference time point in the time period in which each object appears as information indicating the appearance position. Examples of the reference time point include, but are not limited to, the time point when each object first appears in each time period or the time point when each object last appears in each time period.

[0053] The related information acquisition unit 12 can use the appearance position information thus generated as at least a part of the processing target image related information.

[0054] Information indicating the action of each object at each appearance timing (hereinafter, action information) "Action" includes, but is not limited to, laughing, being surprised, crying, talking, running, walking, throwing, kicking, eating, barking, etc. For example, the related information acquisition unit 12 can input images of each object at each appearance timing into a "classifier that classifies the actions of objects shown in images" generated in advance by machine learning, and acquire the output classification results (action information). Then, the related information acquisition unit 12 can use the acquired action information as at least a part of the processing target image related information.

[0055] When the above-mentioned appearance timing information indicates the elapsed time from the beginning or the identification information of the frame image, the related information acquisition unit 12 can input the frame image in which each object appears into a classifier and obtain the classification result.

[0056] On the other hand, if the appearance timing information indicates the time period in which each object appears, the related information acquisition unit 12 can input frame images at a reference time point in the time period in which each object appears to the classifier and acquire the classification result. Examples of the reference time point include, but are not limited to, the time point when each object first appears in each time period or the time point when each object last appears in each time period.

[0057] Information indicating the emotion of each object at each appearance timing (hereinafter referred to as emotion information) "Emotion" includes, but is not limited to, anger, disgust, fear, joy, sadness, surprise, neutrality, etc. Techniques for estimating the emotion of an object (person or animal) based on the facial expression or voice of the object are widely known. Based on this technique, the related information acquisition unit 12 can acquire the emotion estimation results (emotion information) of each object at each appearance timing. Then, the related information acquisition unit 12 can use the acquired emotion information as at least a part of the processing target image related information.

[0058] When the above-mentioned appearance timing information indicates the elapsed time from the beginning or the identification information of the frame image, the related information acquisition unit 12 can estimate the emotion of each object based on the frame image in which each object appears.

[0059] On the other hand, when the appearance timing information indicates the time period in which each object appears, the related information acquisition unit 12 can estimate the emotion of each object based on a frame image at a reference time point in the time period in which each object appears. Examples of the reference time point include, but are not limited to, the time point when each object first appears in each time period or the time point when each object last appears in each time period.

[0060] Information indicating utterance contents arranged in chronological order (hereinafter referred to as utterance content information) The related information acquisition unit 12 can perform speech recognition processing on the processing target image and identify the utterance contents within the processing target image (so-called transcription).The related information acquisition unit 12 can then generate information (utterance content information) in which the identified utterance contents are arranged in chronological order.The related information acquisition unit 12 can use the generated utterance content information as at least a part of the processing target image related information.

[0061] An example of the processing target image related information is shown in Fig. 4. Note that the processing target image related information shown in Fig. 4 is merely an example, and is not limited to this.

[0062] 4 shows an example in which the object is a person. The "person" column shows the name of the person appearing in the image to be processed.

[0063] 4, the image to be processed is divided into a plurality of time periods, each of which is one minute (unit time) from the beginning, and information about each person is shown for each time period.

[0064] 4, multi-dimensional information is shown for each person, linked to each time period. This multi-dimensional information indicates information about each person.

[0065] One of the pieces of multi-dimensional information indicates, for example, whether or not each person appears, and is defined in advance as, for example, "0: appears, 1: does not appear."

[0066] Another piece of multidimensional information indicates, for example, the appearance position of each person in the image.

[0067] Another one of the multiple-dimensional information indicates, for example, the action of each person, which is predefined as, for example, "0: laugh, 1: talk, . . . ".

[0068] Another one of the multiple dimensions of information indicates, for example, the emotion of each person, which is predefined as, for example, "0: anger, 1: disgust, . . . ".

[0069] In this way, the related information acquisition unit 12 can acquire information related to the image to be processed that is obtained by breaking down the image to be processed.

[0070] Returning to FIG. 1 , the analysis unit 13 performs at least one of determining the identity of the processing target image and the reference image and evaluating the object appearing in the processing target image, based on the processing target image related information acquired by the related information acquisition unit 12. As described above, the processing target image related information is information obtained by breaking down the processing target image. By using such processing target image related information to determine the identity and evaluate the object, the accuracy of the analysis is improved. The analysis performed by the analysis unit 13 will be described in detail below.

[0071] Determining Identity Between a Target Image and a Reference Image The target image in this determination is, for example, a video uploaded to a server and made public. The reference image in this determination is legitimate content that is the subject of an investigation to determine whether it has been uploaded to a server and made public without the permission of the rights holder. Examples of legitimate content include, but are not limited to, dramas, movies, television programs, sports programs, music programs, and anime. The legitimate content may also include video in which pages or frames of a manga are extracted and arranged in chronological order.

[0072] The combination of the processing target image and the reference image shown here is merely an example, and the present invention is not limited to this example.

[0073] The related information acquisition unit 12 described above also performs the same processing on the reference image as on the processing target image, generating reference image related information. The reference image related information is information related to the reference image, and indicates the same type of information as the processing target image related information described above. The same type of information as the processing target image related information is at least one of the above-mentioned appearing object information, appearance timing information, total appearance time information, appearance position information, action information, emotion information, and statement content information.

[0074] Then, the analysis unit 13 determines whether the processing target image and the reference image are identical to each other based on the processing target image related information and the reference image related information.

[0075] In one example, when the multiple types of information indicated by the processing target image-related information and the multiple types of information indicated by the reference image-related information completely match, the analysis unit 13 can determine that the processing target image and the reference image are the same. The multiple types of information indicated by the processing target image-related information and the reference image-related information are at least two of the above-mentioned appearing object information, appearance timing information, total appearance time information, appearance position information, action information, emotion information, and statement content information.

[0076] According to this determination example, it is possible to determine with high accuracy whether the processing target image and the reference image are identical.

[0077] However, videos uploaded to a server and made public without the permission of the rights holder may have undergone image processing to avoid detection. As a result, even if the processing target image and the reference image are the same content, a situation may arise in which the processing target image-related information and the reference image-related information do not completely match. Therefore, the analysis unit 13 may be configured to determine that the processing target image and the reference image are the same when the processing target image-related information and the reference image-related information have a certain degree of commonality.

[0078] For example, a criterion may be set in advance that is satisfied when the processing target image-related information and the reference image-related information have a certain degree of commonality. Then, when the processing target image-related information and the reference image-related information satisfy the criterion, the analysis unit 13 may determine that the processing target image and the reference image are the same. Note that there may be only one criterion. Alternatively, multiple criterions may be set. Then, the analysis unit 13 may determine that the processing target image and the reference image are the same when any of the multiple criterions is satisfied.

[0079] The criterion may be any one of the following criteria: The criterion may also be a criterion in which a plurality of the following criteria are connected by an arbitrary logical operator.・The objects appearing in the image to be processed and the objects appearing in the reference image match perfectly. ・A specified percentage or more of the objects appearing in the image to be processed also appear in the reference image. ・A specified percentage or more of the objects appearing in the reference image also appear in the image to be processed. ・The comment content information of the image to be processed and the comment content information of the reference image match perfectly. ・A specified percentage or more of the comment content indicated in the comment content information of the image to be processed is included in the comment content information of the reference image. ・A specified percentage or more of the comment content indicated in the comment content information of the reference image is included in the comment content information of the image to be processed. ・The similarity between the appearance timing information of the image to be processed and the appearance timing information of the reference image is equal to or greater than a standard value. ・The similarity between the appearance position information of the image to be processed and the appearance position information of the reference image is equal to or greater than a standard value. ・The similarity between the action information of the image to be processed and the action information of the reference image is equal to or greater than a standard value. ・The similarity between the emotion information of the image to be processed and the emotion information of the reference image is equal to or greater than a standard value. ・The similarity between the total time information of the image to be processed and the total time information of the reference image is equal to or greater than a standard value.

[0080] The similarity of the appearance timing information can be calculated using, for example, a well-known technique for calculating the similarity of time-series data (numerical array data). For example, the appearance timing information for each object can be expressed using a representation such as "00000111..." to indicate whether or not the object appears in the image at each timing. Each number corresponds to one unit of time, and multiple values ​​are arranged in chronological order. Predefined values ​​such as "0: appears, 1: does not appear" indicate appearance / non-appearance at each unit of time. For example, the analysis unit 13 calculates the similarity between the appearance timing information of the processing target image and the appearance timing information of the reference image for each object. The analysis unit 13 can perform this calculation based on a well-known technique for calculating the similarity between time-series data. The analysis unit 13 can then integrate the similarities for each object using any method and calculate the result as the similarity between the appearance timing information of the processing target image and the appearance timing information of the reference image. Examples of integration methods include, but are not limited to, calculation of statistical values ​​(average, maximum, minimum, mode, median, etc.).

[0081] The analysis unit 13 can calculate the similarity of the appearance position information, the similarity of the action information, and the similarity of the emotion information using the same method as that for the similarity of the appearance timing information.

[0082] The similarity of the total time information can be calculated using any known technology. For example, the analysis unit 13 may calculate the difference or rate of change between the total time information of the processing target image and the total time information of the reference image for each object. The analysis unit 13 may then calculate the similarity of the total time information based on the result of integrating the difference or rate of change for each object using any method. Examples of the integration method include, but are not limited to, calculating statistical values ​​(average, maximum, minimum, mode, median, etc.). A rule (function, table, etc.) for calculating the similarity from the integration result may be generated in advance. The rule is configured to calculate a lower similarity as the difference or rate of change increases. The analysis unit 13 can calculate the similarity of the total time information of the processing target image and the total time information of the reference image based on the integration result and the rule.

[0083] Evaluation of Objects Appearing in Processing Target Images The processing target images in this determination are legitimate content. Examples of legitimate content include, but are not limited to, dramas, movies, television programs, sports programs, music programs, and anime. Legitimate content may also include images of manga extracted by page or frame and arranged in chronological order. Note that the processing target images exemplified here are merely examples and are not limited to these examples.

[0084] The analysis unit 13 calculates the contribution of each object to the content based on the content of the processing target image indicated by the processing target image related information. The analysis unit 13 calculates the contribution of each object to the content based on the "rule for calculating the contribution of each object from the processing target image related information of each object" and the "processing target image related information of each object" that have been generated in advance.

[0085] The rule may be, for example, a function that takes information related to the image to be processed as input and outputs a contribution level, or may be other. The rule may be configured to calculate a higher contribution level as the total appearance time increases. The contribution level may be indicated numerically or by other methods (such as a five-point scale).

[0086] The rule may be configured such that the more times a predetermined point-adding action is performed, the higher the calculated contribution level. Point-adding actions may include reactions such as laughing, being surprised, crying, and speaking.

[0087] The rule may also be configured such that the more times a specific point-adding action is performed immediately after another person performs a specific action, the higher the contribution level is calculated. The specific actions of the other person may include talking, standing up, dancing, etc. The point-adding actions may include reactions such as laughing, being surprised, crying, etc. When this rule is applied, the action information in the processing target image related information may further indicate whether each action was performed within a specific time period after the other person performed the specific action. When the related information acquisition unit 12 analyzes the image and detects a point-adding action of each person, it analyzes whether the other person performed the specific action immediately before that (e.g., during the time period from the detection of the point-adding action until a specific time before). Then, the related information acquisition unit 12 generates the processing target image related information as described above based on the results of the analysis.

[0088] The rule may be configured such that the more times a predetermined emotion is shown, the higher the calculated contribution level. The emotion may include anger, disgust, fear, joy, sadness, surprise, etc.

[0089] Next, an example of the flow of processing executed by the image analyzing device 10 will be described with reference to the flowchart of FIG.

[0090] First, the image analysis device 10 acquires a processing target image (S10). Next, the image analysis device 10 acquires processing target image-related information that indicates at least objects appearing in the processing target image acquired in S10 (S11). Then, based on the processing target image-related information acquired in S11, the image analysis device 10 performs at least one of determining the identity of the processing target image and the reference image and evaluating the objects appearing in the processing target image.

[0091] When the image analysis device 10 determines whether the processing target image and the reference image are identical, it can output the determination result (whether they are identical or not). In this case, the image analysis device 10 may further output the determination criteria that served as the basis for determining that they are identical, i.e., the determination criteria that the processing target image-related information and the reference image-related information satisfied. Based on this output result, the user can understand in what respects the processing target image and the reference image are similar.

[0092] On the other hand, when the image analysis device 10 evaluates objects appearing in the image to be processed, it can output the evaluation results for each object.

[0093] The output of the information as described above can be realized via any output device such as a display, a projection device, a speaker, a printer, or the like.

[0094] "Effects" According to the image analysis device 10 of the second embodiment, the same effects as those of the image analysis device 10 of the first embodiment are achieved.

[0095] Furthermore, the image analysis device 10 can acquire information related to the image to be processed by breaking down the image to be processed using a characteristic method. The information related to the image to be processed includes, for example, the above-mentioned information on the object that appears, information on the timing of appearance, information on the total appearance time, information on the position of appearance, information on action, information on emotion, and information on the content of remarks. The image analysis device 10 then uses this information related to the image to be processed to determine identity and evaluate the object. The image analysis device 10 improves the accuracy of the identity determination and evaluation of the object.

[0096] Furthermore, the image analyzing device 10 can determine whether the processing target image and the reference image are identical based on the above-described characteristic criteria. With this image analyzing device 10, it is possible to accurately determine whether the processing target image and the reference image are identical, even if the processing target image has been subjected to image processing.

[0097] Furthermore, the image analysis device 10 can calculate the contribution of an object appearing in legitimate content (the image to be processed) to the content using the characteristic method described above. Based on the calculated contribution, the user can decide who will participate in the next content creation, provide feedback to each person, and make various improvements.

[0098] <Modifications> Modifications applicable to the first and second embodiments will be described. Even when these modifications are adopted, the same effects as those of the first and second embodiments can be achieved.

[0099] "Modification 1" Modification 1 can be applied to a process for evaluating an object that appears in an image to be processed. In this case, the legitimate content (image to be processed) is content broadcast on television or the Internet.

[0100] The related information acquisition unit 12 acquires information related to the image to be processed, which further indicates an analysis result of the posted information posted on social media at the time the image to be processed was broadcast. For example, the related information acquisition unit 12 acquires posted information posted in relation to each legitimate content using the content name, broadcast time, cast, etc. of the legitimate content as keys. The related information acquisition unit 12 can then analyze the acquired posted information using any widely known technology. The analysis result may be, for example, the number of posted information items posted at each time, but is not limited to this.

[0101] The analysis unit 13 uses such processing target image related information to evaluate the object appearing in the processing target image.

[0102] The analysis unit 13 may calculate the contribution of a target person to be high if the number of pieces of posted information at the time when the target person appears or immediately thereafter (for example, a time period from the time when the target person appears until a predetermined time later) is equal to or greater than a threshold. Furthermore, the analysis unit 13 may calculate the contribution of a target person to be high if the number of pieces of posted information increases at the time when the target person appears or immediately thereafter (for example, a time period from the time when the target person appears until a predetermined time later).

[0103] Additionally, the analysis unit 13 may calculate the contribution level of a target person to be high if the number of pieces of posted information at the time when the target person performed a predetermined action or immediately thereafter is equal to or greater than a threshold. Furthermore, the analysis unit 13 may calculate the contribution level of a target person to be high if the number of pieces of posted information increases at the time when the target person performed a predetermined action or immediately thereafter.

[0104] Additionally, the analysis unit 13 may calculate the contribution level of a target person to be high if the number of pieces of posted information at or immediately after the target person shows a predetermined emotion is equal to or greater than a threshold. Furthermore, the analysis unit 13 may calculate the contribution level of a target person to be high if the number of pieces of posted information increases at or immediately after the target person shows a predetermined emotion.

[0105] According to the first modification, the degree of contribution of the target person to the legitimate content can be calculated with higher accuracy.

[0106] "Variation 2" At least some of the multiple types of information indicated by the processing target image related information may be generated by a user visually checking the processing target image, understanding its contents, and inputting the understood contents into the image analysis device 10.

[0107] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0108] In addition, in the flowcharts used in the above explanation, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content.

[0109] Some or all of the above embodiments may be described as, but are not limited to, the following notes. 1. An image analysis device comprising: an image acquisition means for acquiring a processing target image; related information acquisition means for acquiring processing target image related information indicating at least objects appearing in the processing target image; and analysis means for performing at least one of determining the identity of the processing target image and a reference image and evaluating the objects appearing in the processing target image based on the processing target image related information. 2. The image analysis device described in 1, wherein the processing target image related information further indicates at least one of the appearance timing of each of the objects, the total appearance time of each of the objects, the appearance position in the image at each appearance timing of each of the objects, the action at each appearance timing of each of the objects, and the emotion at each appearance timing of each of the objects. 3. The image analysis device described in 1, wherein the processing target image related information further indicates utterance content arranged in chronological order. 4. The image analyzing device described in 2 or 3, wherein the analysis means determines the identity of the image to be processed and the reference image based on the information related to the image to be processed and reference image related information that indicates at least the object appearing in the reference image. 5. The image analyzing device described in 4, wherein the image to be processed is an image that has been uploaded to a server and made public, and the reference image is legitimate content. 6. The image analyzing device described in 1, wherein the image to be processed is content, and the analysis means calculates the degree of contribution of each of the objects to the content based on the content of the image to be processed indicated in the information related to the image to be processed. 7. The image analyzing device described in 6, wherein the analysis means calculates the degree of contribution of each of the objects to the content based on a predetermined point-adding action performed by each of the objects. 8. The image analyzing device described in 6 or 7, wherein the image to be processed is broadcast on television or the Internet, and the information related to the image to be processed further includes the results of analysis of information posted on social media at the time the image to be processed was broadcast.9. An image analysis method in which one or more computers acquire a processing target image, acquire processing target image related information that indicates at least an object appearing in the processing target image, and perform at least one of determining the identity of the processing target image and a reference image and evaluating the object appearing in the processing target image based on the processing target image related information. 10. A program that causes a computer to function as: image acquisition means that acquires a processing target image, related information acquisition means that acquires processing target image related information that indicates at least an object appearing in the processing target image, and analysis means that perform at least one of determining the identity of the processing target image and a reference image and evaluating the object appearing in the processing target image based on the processing target image related information.

[0110] Some or all of Supplements 2 to 8 that are dependent on the image analysis device of Supplement 1 described above may also be dependent on the image analysis method of Supplement 9 and the program of Supplement 10 in the same dependent relationship as Supplement 1 and Supplements 2 to 8. Furthermore, within the scope of each of the above-mentioned embodiments, some or all of the configurations described as Supplements can be realized in various hardware, software, various recording means for recording software, or systems.

[0111] This application claims priority based on Japanese Patent Application No. 2023-122248, filed on July 27, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0112] 10 Image analysis device 11 Image acquisition unit 12 Related information acquisition unit 13 Analysis unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. An image analysis device having an image acquisition means for acquiring an image to be processed, a related information acquisition means for acquiring information related to the image to be processed that indicates at least objects appearing in the image to be processed, and an analysis means for performing at least one of determining the identity of the image to be processed and a reference image and evaluating the objects appearing in the image to be processed based on the information related to the image to be processed.

2. The image analysis device of claim 1, wherein the information related to the image to be processed further indicates at least one of the following: the appearance timing of each of the objects; the total appearance time of each of the objects; the appearance position within the image at each appearance timing of each of the objects; the action at each appearance timing of each of the objects; and the emotion at each appearance timing of each of the objects.

3. The image analysis device according to claim 1 or 2, wherein the image-related information to be processed further indicates the contents of comments arranged in chronological order.

4. An image analysis device as described in claim 2 or 3, wherein the analysis means determines the identity of the image to be processed and the reference image based on information related to the image to be processed and information related to a reference image indicating at least the object appearing in the reference image.

5. The image analyzing device according to claim 4, wherein the image to be processed is an image uploaded to a server and made public, and the reference image is legitimate content.

6. An image analysis device as described in any one of claims 1 to 5, wherein the image to be processed is content, and the analysis means calculates the contribution of each of the objects to the content based on the content of the image to be processed indicated in the information related to the image to be processed.

7. The image analysis device according to claim 6, wherein the analysis means calculates the contribution of each of the objects to the content based on a predetermined point-adding action performed by each of the objects.

8. An image analyzing device as described in claim 6 or 7, wherein the image to be processed is broadcast on television or the Internet, and the information related to the image to be processed further includes an analysis result of posted information on social media at the time the image to be processed was broadcast.

9. An image analysis method in which one or more computers acquire an image to be processed, acquire information related to the image to be processed that indicates at least objects that appear in the image to be processed, and perform at least one of determining the identity of the image to be processed and a reference image and evaluating the objects that appear in the image to be processed based on the information related to the image to be processed.

10. The image analysis method of claim 9, wherein the information related to the image to be processed further indicates at least one of the following: appearance timing of each of the objects; total appearance time of each of the objects; appearance position within the image at each appearance timing of each of the objects; action at each appearance timing of each of the objects; and emotion at each appearance timing of each of the objects.

11. The image analysis method according to claim 9 or 10, wherein the information related to the image to be processed further indicates the contents of the comments arranged in chronological order.

12. An image analysis method as described in claim 10 or 11, in which the one or more computers determine the identity of the image to be processed and the reference image based on information related to the image to be processed and information related to a reference image that indicates at least the object appearing in the reference image.

13. The image analysis method according to claim 12, wherein the image to be processed is an image uploaded to a server and made public, and the reference image is legitimate content.

14. An image analysis method according to any one of claims 9 to 13, wherein the image to be processed is content, and the one or more computers calculate the contribution of each of the objects to the content based on the content of the image to be processed indicated in the information related to the image to be processed.

15. A recording medium having recorded thereon a program that causes a computer to function as: an image acquisition means for acquiring an image to be processed; a related information acquisition means for acquiring information related to the image to be processed that indicates at least objects that appear in the image to be processed; and an analysis means for performing at least one of determining the identity of the image to be processed and a reference image based on the information related to the image to be processed, and evaluating the objects that appear in the image to be processed.

16. The recording medium described in claim 15, wherein the information related to the image to be processed further indicates at least one of the following: appearance timing of each of the objects; total appearance time of each of the objects; appearance position within the image at each appearance timing of each of the objects; action at each appearance timing of each of the objects; and emotion at each appearance timing of each of the objects.

17. The recording medium according to claim 15 or 16, wherein the information related to the image to be processed further indicates the contents of the comments arranged in chronological order.

18. A recording medium as described in claim 16 or 17, wherein the analysis means determines the identity of the image to be processed and the reference image based on information related to the image to be processed and information related to a reference image indicating at least the object appearing in the reference image.

19. The recording medium according to claim 18, wherein the image to be processed is an image uploaded to a server and made public, and the reference image is regular content.

20. A recording medium described in any one of claims 15 to 19, wherein the image to be processed is content, and the analysis means calculates the contribution of each of the objects to the content based on the content of the image to be processed indicated in the information related to the image to be processed.

Images