Image analysis device, image analysis method, and recording medium
The image analysis device and method address the challenge of determining relationships between individuals from videos by processing attribute information to identify related persons and visualize their connections, enhancing relationship understanding and investigation efficiency.
Patent Information
- Application Number
- PCT/JP2024/016649
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-01
- Filing Date
- 2024-04-30
- Publication Date
- 2024-11-07
AI Technical Summary
Existing image analysis technologies cannot effectively determine the relationship between a specific person identified from a video and individuals related to that person, lacking the ability to provide clear organizational and interpersonal connections.
An image analysis device and method that acquires attribute information from multiple images, accepts person designation inputs, and outputs screens displaying relationship and organizational information between the designated person and related individuals, using attribute information to identify related persons based on time spent and frequency of appearance, and visualizing this information through organizational charts or marks.
Enables easy understanding of relationships between individuals identified from videos, improving efficiency in tasks such as command chain analysis and group crime investigations by providing detailed interpersonal and organizational connections.
Smart Images

Figure JP2024016649_07112024_PF_FP_ABST
Abstract
Description
Image analysis device, image analysis method, and recording medium
[0001] The present invention relates to an image analysis device, an image analysis method, and a recording medium.
[0002] Techniques for analyzing images to identify specific people are used in a variety of situations.
[0003] An example of a technology for identifying a specific person by analyzing an image is disclosed in the following Patent Document 1. Patent Document 1 discloses a technology for identifying a specific person and their potential companions based on an input video. For example, Patent Document 1 discloses a method for determining potential companions for a specific person based on the number of times the specific person and other people appear together.
[0004] Japanese Patent Application Laid-Open No. 2023-018113
[0005] When a specific person and a person related to that person are identified from a video, there is a need to understand the relationship between those people. The technology of Cited Document 1 does not determine this relationship.
[0006] In view of the above-mentioned problems, one example of the object of the present invention is to provide an image analysis device, an image analysis method, and a program (a recording medium for recording the program) that make it possible to easily understand the relationship between a specific person identified from video and people related to that person.
[0007] The image analysis device of the present disclosure comprises: an attribute information acquisition means for processing a plurality of images to acquire attribute information indicating the attributes of each person appearing in each image; a designated input receiving means for receiving a person designation input indicating a designated person; a related person identification means for identifying a related person who is a person related to the designated person using the attribute information acquired for each person; and a display output means for outputting a screen including person relationship information indicating the relationship between the designated person and the related person, and organizational information regarding the organizations to which the designated person and the related person each belong.
[0008] The image analysis method of the present disclosure includes at least one computer processing a plurality of images to obtain attribute information indicating the attributes of each person appearing in each image; accepting a person designation input indicating a designated person; identifying associated persons who are persons related to the designated person using the attribute information obtained for each person; and outputting a screen including person relationship information indicating the relationship between the designated person and the associated persons, and organizational information regarding organizations to which the designated person and the associated persons each belong.
[0009] The recording medium of the present disclosure records a program that, when executed by at least one computer, causes the computer to perform operations including: processing a plurality of images to obtain attribute information indicating the attributes of each person appearing in each image; accepting a person designation input indicating a designated person; identifying associated persons who are persons related to the designated person using the attribute information obtained for each person; and outputting a screen including person relationship information indicating the relationship between the designated person and the associated person, and organizational information regarding organizations to which the designated person and the associated person each belong.
[0010] According to one aspect of the present invention, for example, a person who is checking a video can easily understand the relationships between multiple people identified from the video.
[0011] 1 is a diagram illustrating an overview of an image analyzing device according to a first embodiment. FIG. 1 is a diagram illustrating an example of the system configuration of an image analysis system configured to include an image analyzing device. FIG. 2 is a diagram illustrating an example of information stored in a database. FIG. 3 is a diagram illustrating an example of the hardware configuration of a computer that realizes an image analyzing device. FIG. 4 is a flowchart illustrating an example of processing executed by an image analyzing device according to a second embodiment. FIG. 5 is a diagram illustrating an example of a screen output by a display output unit. FIG. 6 is a diagram illustrating another example of a screen output by a display output unit. FIG. 7 is a diagram for explaining the operation of an image analyzing device (display output unit) according to a first modified example of the second embodiment. FIG. 8 is a diagram for explaining the operation of an image analyzing device (display output unit) according to a first modified example of the second embodiment. FIG. 9 is a diagram for explaining the operation of an image analyzing device (associated person identification unit and display output unit) according to a second modified example of the second embodiment. FIG. 10 is a diagram illustrating an example of a screen updated in response to a criterion change input. FIG. 11 is a diagram for explaining the operation of an image analyzing device (associated person identification unit and display output unit) according to a third modified example of the second embodiment. FIG. 12 is a diagram illustrating an example of a screen updated in response to a criterion change input. FIG. 13 is a diagram illustrating the system configuration of an image analysis system 1 according to a third embodiment. FIG. 14 is a diagram illustrating an example of information stored in a posture information database. FIG. 10 is a diagram for explaining the operation of the image analyzing device (associated person identifying unit and display output unit) according to the third embodiment. FIG. 11 is a diagram for explaining the operation of the image analyzing device (associated person identifying unit and display output unit) according to the third embodiment. FIG. 12 is a diagram for explaining the operation of the image analyzing device (associated person identifying unit and display output unit) according to the third embodiment.
[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted where appropriate. Furthermore, unless otherwise specified, each block in each block diagram represents a functional configuration, not a hardware configuration. Furthermore, when elements such as blocks in a diagram are linked by arrows, the direction of the arrows is merely intended to make the flow of information easier to understand. In this case, the direction of the arrows does not limit the direction of communication (one-way communication / two-way communication) unless otherwise specified.
[0013] In the following description, "acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition), and a device inputting data or information output from another device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a reply, and accessing and reading information from another device or storage medium. Examples of passive acquisition include receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.
[0014] 1 is a diagram showing an overview of an image analysis device 10 according to a first embodiment. The image analysis device 10 shown in the diagram includes an attribute information acquisition unit 110, a designated input acceptance unit 120, a related person identification unit 130, and a display output unit 140.
[0015] The attribute information acquisition unit 110 processes multiple input images. Based on the processing results of each image, the attribute information acquisition unit 110 acquires attribute information indicating the attributes of each person appearing in each image. The designated input acceptance unit 120 accepts a person designation input indicating a certain person. In the following description, a person designated by the person designation input will also be referred to as a "designated person." The associated person identification unit 130 uses the attribute information acquired for each person to identify people associated with the designated person. In the following description, a person associated with the designated person will also be referred to as an "associated person." The display output unit 140 outputs a screen including person-related information and organizational information to an output device such as a display. Here, "person-related information" refers to information indicating the relationship between the designated person and associated people. Furthermore, "organization information" refers to information regarding the organizations to which the designated person and associated people belong. Specific examples of person-related information and organizational information will be described later.
[0016] According to this embodiment, attribute information for each person is acquired for each image by processing each of a plurality of images. When a person is identified as a designated person, people related to the designated person are identified based on the attribute information acquired for each image. A screen is then output, including personal relationship information indicating the relationship between the designated person and the specific person and organizational information regarding the organization to which each person belongs. In other words, an image analysis device and an image analysis method are provided that enable easy understanding of the relationship between a specific person and people related to that person. Furthermore, as will be understood from the description of the second embodiment described below, a program for realizing the functions of the image analysis device 10, which is a means for solving the above problem, on at least one computer, and a non-transitory computer-readable recording medium for storing the program are also provided.
[0017] A detailed example of the image analysis device 10 in another embodiment will be described below.
[0018] Second Embodiment FIG. 2 is a diagram illustrating an example of the system configuration of an image analysis system 1 including an image analysis device 10. The image analysis system 1 illustrated in FIG. 2 includes the image analysis device 10, a database 20, and a display 30. The image analysis device 10 and the database 20 are connected via a network 40. The display 30 serves as a display for the image analysis device 10 and is connected via an input interface (not shown) of the image analysis device 10. The configuration of the image analysis system 1 is not limited to the example shown in this figure. For example, the database 20 may be constructed in a storage area inside the image analysis device 10. For example, the display 30 may be a display of a computer (not shown) connected to the image analysis device 10 via the network 40.
[0019] As illustrated in FIG. 3 , the database 20 stores various information related to various people. FIG. 3 is a diagram illustrating an example of information stored in the database 20. In the example of FIG. 3 , the database 20 stores registration information for face recognition, the person's name, personal attributes, and affiliated organization information. Here, the registration information for face recognition includes at least one of a face image and a feature extractable based on the face image. Furthermore, the personal attributes include any attribute other than facial features that can be used to identify at least one person from an image. For example, the personal attributes include information related to a person's physical characteristics (a combination of various features such as skeletal shape, height, body size, hair color, and skin color). In addition to these optional attributes, the personal attributes may also include other attributes such as age and gender. Furthermore, the affiliated organization information includes information identifying the organization to which the target person belongs, information indicating the person's position and rank within the organization, and the like. Furthermore, the affiliated organization information may also include information indicating the system (structure) of the relevant organization.
[0020] Information such as that illustrated in FIG. 3 is accumulated, for example, as follows: First, the image analyzing device 10 or a management terminal (not shown) receives as input a facial image of a target person, as well as information about the person's attributes and information about the person's affiliated organization. The image analyzing device 10 or a management terminal (not shown) then compiles the information received as input to generate a single record. The image analyzing device 10 or a management terminal (not shown) then stores the generated record in the database 20. In this manner, information such as that illustrated in FIG. 3 is accumulated in the database 20.
[0021] The image analyzing device 10 includes at least the attribute information acquiring unit 110, the designated input receiving unit 120, the related person identifying unit 130, and the display output unit 140 described in the first embodiment.
[0022] FIG. 4 is a diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the image analyzing apparatus 10.
[0023] The computer 1000 has a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .
[0024] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.
[0025] The processor 1020 is a central processing unit (CPU) or a graphics processing unit (GPU).
[0026] The memory 1030 is a main storage device realized by a random access memory (RAM) or the like.
[0027] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), etc. The storage device 1040 stores at least program modules that realize the functions of the image analysis system 1 described above (the attribute information acquisition unit 110, the designated input acceptance unit 120, the related person identification unit 130, and the display output unit 140). The storage device 1040 may also function as the database 20.
[0028] The processor 1020 loads a program module read from the storage device 1040 onto the memory 1030 and executes the program module, thereby realizing the function corresponding to the program module. For example, the processor 1020 reads a program module corresponding to the attribute information acquisition unit 110 onto the memory 1030 and executes the program module, thereby realizing the function of the attribute information acquisition unit 110 described in the present disclosure. Similarly, the processor 1020 reads a program module corresponding to the designated input acceptance unit 120 onto the memory 1030 and executes the program module, thereby realizing the function of the designated input acceptance unit 120 described in the present disclosure. Similarly, the processor 1020 reads a program module corresponding to the associated person identification unit 130 onto the memory 1030 and executes the program module, thereby realizing the function of the associated person identification unit 130 described in the present disclosure. Similarly, the processor 1020 reads a program module corresponding to the display output unit 140 onto the memory 1030 and executes the program module, thereby realizing the function of the display output unit 140 described in the present disclosure. The operation of the processor 1020 is common to each embodiment included in the present disclosure.
[0029] The above-described program modules may be recorded on a recording medium other than the storage device 1040. The recording medium on which the program modules are recorded includes any non-transitory, tangible medium usable by the computer 1000. The recording medium on which the program modules are recorded may have embedded therein program code that is readable by the computer 1000 (processor 1020).
[0030] The input / output interface 1050 is an interface for connecting the computer 1000 (image analyzing device 10) to various input / output devices. For example, an output device such as a display 30 is connected to the computer 1000 (image analyzing device 10) via the input / output interface 1050.
[0031] The network interface 1060 is an interface for connecting the computer 1000 (image analysis device 10) to a network. This network includes, for example, a local area network (LAN) and a wide area network (WAN). The network interface 1060 connects the computer 1000 (image analysis device 10) to various networks via a wireless or wired connection. The computer 1000 (image analysis device 10) can communicate with, for example, a database 20 via the network interface 1060.
[0032] The computer 1000 (image analyzing device 10) can also be connected to various devices (not shown) via the input / output interface 1050 or the network interface 1060. The "various devices" referred to here are not particularly limited, but include, for example, input devices such as operation buttons, keyboards, touch panels, and mice, and output devices such as speakers, microphones, and printers.
[0033] FIG. 5 is a flowchart illustrating a process executed by the image analyzing device 10 according to the second embodiment.
[0034] First, the attribute information acquisition unit 110 acquires a plurality of images (e.g., video data) to be processed and analyzes each image (step S102). As a result of the analysis process for each image, the attribute information acquisition unit 110 acquires attribute information of each person for each image.
[0035] As an example, the attribute information acquisition unit 110 acquires, as attribute information for each person, facial images of each person or facial feature values based on the facial images, full-body images of each person or skeletal information based on the full-body images, etc. In this case, the attribute information acquisition unit 110 first identifies image areas (hereinafter also referred to as "person areas") corresponding to each person appearing in multiple images to be processed. The attribute information acquisition unit 110 generates skeletal information for each person from the identified person areas for each person. The attribute information acquisition unit 110 can generate skeletal information from the person areas using a known image processing algorithm (e.g., an image processing algorithm for identifying a person's joints from an image of a person). The attribute information acquisition unit 110 can further extract a face area (hereinafter also referred to as a "face area") from the person area. The attribute information acquisition unit 110 can extract the face area of each person using a known face recognition algorithm. In this case, the attribute information acquisition unit 110 may generate facial feature information from the face areas of each person. Then, the attribute information acquisition unit 110 acquires the information obtained about each person (such as face area, facial feature information, and skeletal information) as attribute information for each person. The attribute information for each person acquired by the attribute information acquisition unit 110 is compared with information about each person in the database 20. As a result of the comparison, each person appearing in each image is identified. The identification results for each person are stored in a storage area of the image analysis device 10, such as the memory 1030 or the storage device 1040.
[0036] The image analysis device 10 may further have a function to generate a list of identified people based on the identification results of each person and output the list to the display 30. The image analysis device 10 may further have a function to perform a process of adding information indicating the area of each person (e.g., a rectangular frame) to each image and output a screen including the processed image to the display 30.
[0037] Next, the designated input accepting unit 120 acquires a person designation input (step S104). For example, the designated input accepting unit 120 accepts an input operation of information identifying the designated person via an input device connected to the input / output interface 1050. The input operation includes, for example, an operation of selecting the designated person from a person list displayed on the display 30, an operation of selecting an area corresponding to the designated person within an image displayed on the display 30, etc. The designated input accepting unit 120 may also accept another image (query image) related to the designated person as input.
[0038] Next, the related person identification unit 130 identifies related people to the specified person indicated by the person designation input (step S106). The related person identification unit 130 identifies related people to the specified person using the attribute information of each person acquired from each image in the processing of step S102.
[0039] For example, the associated person identification unit 130 can identify an associated person as follows. First, the associated person identification unit 130 calculates at least one of the time spent with the designated person and the number of times the person appears with the designated person, based on the attribute information of each person acquired from each image. For example, suppose that the attribute information acquisition unit 110 processes video data (multiple images) shot at 30 fps, and as a result, attribute information of a first person corresponding to the designated person and attribute information of a second person different from the designated person are simultaneously acquired in three sections within the video. Furthermore, suppose that the three sections are a first section (300 frames), a second section (900 frames), and a third section (600 frames). In this case, the associated person identification unit 130 can calculate the time spent together with the first person and the second person by adding up the times corresponding to each section. That is, the associated person identification unit 130 can calculate the time that the first person (designated person) and the second person are together as "60 seconds" by adding up the time "10 seconds" corresponding to the first section, the time "30 seconds" corresponding to the second section, and the time "20 seconds" corresponding to the third section. The associated person identification unit 130 can also calculate the number of sections in the video, "3," as the number of times that the first person (designated person) and the second person appear together. The associated person identification unit 130 then compares at least one of the calculated time spent with the first person (designated person) and the number of times that the second person appears with the first person (designated person) with a predetermined criterion for determining an associated person. The criterion defines at least one of a reference time for determining an associated person and a number of appearances for determining an associated person. The criterion is also stored in advance in a storage area accessible to the associated person identification unit 130, such as the storage device 1040. For example, if the length of time calculated for a second person is equal to or greater than a predetermined reference time, the associated person identification unit 130 may identify the second person as an associated person. Also, for example, if the number of appearances calculated for a second person is equal to or greater than a predetermined reference number, the associated person identification unit 130 may identify the second person as an associated person.For example, if both the time and the number of appearances calculated for a second person exceed the standards set for each, the related person identification unit 130 may identify the second person as a related person.
[0040] Next, the display output unit 140 generates personal relationship information and organizational information indicating the relationship between the specified person and related people (step S108).Then, the display output unit 140 outputs a screen including the personal relationship information and organizational information to the display 30 (step S110).For example, the display output unit 140 performs the following operation.
[0041] First, the display output unit 140 reads from the database 20 organizational information of the designated person identified by the person designation input received by the designation input receiving unit 120. The display output unit 140 also reads from the database 20 organizational information of the person identified as the related person by the related person identification unit 130. Here, assume that the organizational information of the designated person indicates a first organization, and the organizational information of the related person indicates a second organization different from the first organization. In this case, the display output unit 140 acquires an organizational chart of the first organization to which the designated person belongs (first organizational chart) and an organizational chart of the second organization to which the related person belongs (second organizational chart). Note that data related to the organizational charts of each organization is pre-stored in a storage area accessible to the display output unit 140, such as the storage device 1040. After acquiring the data of the first organizational chart and the second organizational chart from the storage area, the display output unit 140 generates rendering data for a screen including the first organizational chart and the second organizational chart as the organizational information.
[0042] Furthermore, the display output unit 140 generates personal relationship information indicating the relationship between the designated person and related persons using the first organizational chart and the second organizational chart. For example, the display output unit 140 identifies a portion of the first organizational chart corresponding to the designated person (a position on the organizational chart determined by job title, rank, etc.) based on the organizational information acquired about the designated person. The display output unit 140 also identifies a portion of the second organizational chart corresponding to the related person (a position on the organizational chart determined by job title, rank, etc.) based on the organizational information acquired about the related person. The display output unit 140 then generates personal relationship information between the two organizational charts using the position of the designated person on the first organizational chart and the position of the related person on the second organizational chart. The display output unit 140 combines the generated person identification information with the screen drawing data and outputs the combined information to the display 30.
[0043] As an example, the display output unit 140 outputs a screen as illustrated in FIG. 6 to the display 30. FIG. 6 is a diagram showing an example of a screen output by the display output unit 140. The screen illustrated in this figure includes a line that associates the position of a designated person (person H) with the positions of related people (persons C and F) between the first organizational chart and the second organizational chart. Such a line visualizes the relationship between the designated person and the related people. For example, in the example of this figure, the state in which persons C and F are identified as people related to designated person H is visualized by two lines L1 and L2. By utilizing such a screen, it is possible to easily understand the inter-organizational relationship between the designated person and the related people identified from the analysis results of the input video.
[0044] As another example, the display output unit 140 outputs a screen such as that illustrated in FIG. 7 to the display 30. FIG. 7 is a diagram illustrating another example of a screen output by the display output unit 140. The screen illustrated in this figure includes a star-shaped mark as a common mark for associating a designated person with a related person between the first organizational chart and the second organizational chart. The common mark may be any mark as long as it identifies the correspondence between the designated person and the related person. By assigning a common mark to the designated person and the related person in this manner, the relationship between the designated person and the related person is visualized. For example, in the example illustrated in this figure, the state in which person C and person F are identified as people related to designated person H is visualized by the star-shaped marks assigned to the positions of each person. By utilizing such a screen, it is possible to easily understand the inter-organizational relationship between the designated person and the related person identified from the analysis results of the input video.
[0045] As another example, the display output unit 140 outputs a screen such as that illustrated in FIG. 8 to the display 30. FIG. 8 is a diagram illustrating another example of a screen output by the display output unit 140. For the screen illustrated in this figure, the display output unit 140 sets a common display mode (common background pattern) for a portion of the first organizational chart corresponding to the designated person and a second portion of the second organizational chart corresponding to the related person. The common display mode may be any as long as it allows identification of the correspondence between the designated person and the related person. By utilizing such a screen, it is possible to easily understand the inter-organizational relationship between the designated person and the related person identified from the analysis results of the input video.
[0046] The display output unit 140 may be configured to output to the display 30 a screen that combines two or more of the display examples of FIGS.
[0047] As described above, according to this embodiment, a screen including personal relationship information indicating the relationship between the designated person and the associated person, and organizational information regarding the organizations to which the designated person and the associated person respectively belong, is output to the display 30. This screen allows the person (operator) tasked with reviewing images to understand the relationship between the designated person and the associated person in more detail. For example, by checking the screen output by the display output unit 140, the operator can easily understand the chain of command within and between organizations. Furthermore, for example, if the operator is a person involved in an investigation, the screen output by the display output unit 140 can improve the efficiency of a group crime investigation involving multiple members.
[0048] Modification of the Second Embodiment As a first modification, the display output unit 140 may be configured to output additional information corresponding to the selected personal relationship information when receiving an input to select personal relationship information on a screen such as those illustrated in FIGS. 6 to 8 . As an example, the display output unit 140 receives an input to select a portion on the screen corresponding to the personal relationship information. The display output unit 140 then identifies a designated person and related people indicated by the selected personal relationship information. The display output unit 140 then identifies one or more images in which the designated person and related people appear together from among the multiple images. The display output unit 140 then displays the identified one or more images on the display 30.
[0049] The operation of the image analyzing device 10 (display output unit 140) according to the first modified example will be described with reference to the drawings. FIGS. 9 and 10 are diagrams for explaining the operation of the image analyzing device 10 (display output unit 140) according to the first modified example of the second embodiment. First, the display output unit 140 accepts an input to select a portion (e.g., line L1 / line L2) corresponding to person relationship information on the screen displayed on the display 30, as illustrated in FIG. 9 . This figure illustrates a state in which line L1 corresponding to person relationship information indicating the relationship between a designated person (person H) and a first related person (person C) is selected. In response to the selection input illustrated in FIG. 9 , the display output unit 140 further outputs a screen, for example, as illustrated in FIG. 10 . The screen illustrated in FIG. 10 includes an image showing both person H and person C and information indicating the areas of each person within the image. The image included in the screen illustrated in FIG. 10 is a representative image selected from multiple images processed by the attribute information acquisition unit 110. For example, the display output unit 140 can select an image in which each person is most prominent as a representative image. Furthermore, as illustrated in Fig. 10, the display output unit 140 may further include a button on the screen for switching to another image. Furthermore, the image included on the screen illustrated in Fig. 10 may be partial video data obtained by cutting out a portion of the original video data in which each person is visible together.
[0050] According to the first modification, a person checking the video can easily view an actual image in which the designated person and related people appear together.
[0051] As a second modified example, the display output unit 140 may be configured to output, on the screen, an element for changing the criteria used by the associated person identification unit 130 to identify an associated person. In this case, the display output unit 140 outputs, for example, a screen such as that shown in FIG. 11 to the display 30. FIG. 11 is a diagram for explaining the operation of the image analyzing device 10 (the associated person identification unit 130 and the display output unit 140) according to the second modified example of the second embodiment. The screen illustrated in FIG. 11 includes, in the upper right portion, an element E for setting and changing the criteria for identifying an associated person. The user performs an input on element E to change the criteria related to at least one of the time and the number of times, for example, via an input device connected to the input / output interface 1050.
[0052] When an input for changing the criteria is made in element E illustrated in FIG. 11 , the associated person identification unit 130 updates the identification result of associated persons for the designated person based on the new criteria changed by the input. For example, in the screen illustrated in FIG. 11 , two associated persons (person C and person F) are identified for the designated person (person H). Here, based on the processing results of each image by the attribute information acquisition unit 110 (attribute information of each person acquired from each image), the time that person C was with person H and the number of times that person C appeared with person H are calculated to be "100 seconds" and "3 times," respectively. Similarly, the time that person F was with person H and the number of times that person F appeared with person H are calculated to be "200 seconds" and "5 times," respectively. Then, assume that an input for changing the reference time to "200 seconds" or the reference number of times to "5 times" is made in element E on the screen illustrated in FIG. 11 . In this case, person C will no longer satisfy the new criteria changed by the input. As a result, the associated person identification unit 130 excludes person C from the associated persons and updates the screen display, for example, as shown in Fig. 12. Fig. 12 is a diagram illustrating an example of a screen updated in response to the criteria change input. Unlike the screen in Fig. 11, the screen in Fig. 12 only contains the person relationship information (line L2) indicating the relationship between person H (specified person) and person F (associated person).
[0053] As explained in the second modified example, by making it possible to change the criteria for identifying related persons, it is possible to more flexibly identify persons related to a specified person.
[0054] As a third variation, the associated person identification unit 130 may be configured to identify associated persons by further using the "distance between persons." In this case, the attribute information acquisition unit 110 further acquires, for example, the position of each person (a position in the image coordinate system, or a position in a real-world coordinate system corresponding to the position in the image coordinate system) as an analysis result of the image to be processed. The attribute information acquisition unit 110 can use various known methods to calculate the positions of objects in the image. The associated person identification unit 130 can then calculate the distance between the designated person and each of the other persons based on information about the position of each person acquired from each image. For example, the associated person identification unit 130 calculates the distance to the designated person (average distance) based on position information acquired from each image for each person who is with the designated person. Then, if the calculated distance for a certain person satisfies a predetermined criterion (e.g., the calculated distance is equal to or less than the criterion distance), the associated person identification unit 130 identifies the person as an "associated person."
[0055] In this case, the display output unit 140 may be configured to output a screen such as that shown in FIG. 13 . FIG. 13 is a diagram for explaining the operation of the image analysis device 10 (the associated person identification unit 130 and the display output unit 140) according to a third modified example of the second embodiment. The screen shown in FIG. 13 includes an element E in the upper right portion for setting and changing criteria for identifying associated people. Unlike the diagrams according to the second modified example ( FIGS. 11 and 12 ), the element E in this diagram is configured to be able to accept input for changing the criteria related to "distance." For example, a user inputs an input for changing the criteria related to at least one of time, number of times, and distance on the element E via an input device connected to the input / output interface 1050.
[0056] When an input for changing the criteria is made in element E illustrated in FIG. 13 , the associated person identification unit 130 updates the identification results of associated persons for the designated person based on the new criteria changed by the input. For example, in the screen illustrated in FIG. 13 , two associated persons (persons C and F) are identified for the designated person (person H). Here, based on the processing results of each image by the attribute information acquisition unit 110 (attribute information of each person acquired from each image), the time that person C spent with person H, the number of times that person C appeared with person H, and the distance when person C was with person H are calculated to be "100 seconds," "3 times," and "0.5 m," respectively. Similarly, the time that person F spent with person H, the number of times that person F appeared with person H, and the distance when person F was with person H are calculated to be "200 seconds," "5 times," and "1.0 m," respectively. Then, assume that an input for changing the reference distance to "0.5 m" is made in element E on the screen illustrated in FIG. 13 . In this case, person F no longer satisfies the new criteria changed by the input. As a result, the associated person identification unit 130 excludes person F from the associated persons. As a result, the display output unit 140 updates the screen display, for example, as shown in FIG. 14. FIG. 14 is a diagram illustrating an example of a screen updated in response to the criteria change input. Unlike the screen in FIG. 13, the screen in FIG. 14 only contains person relationship information (line L1) indicating the relationship between person H (specified person) and person C (associated person).
[0057] According to the third modified example, a new index called "distance between people" is used to accurately identify people who are likely to be related to the designated person.
[0058] The image analyzing device 10 may have a combination of the configuration according to the first modified example and the configuration according to the second modified example or the configuration according to the third modified example.
[0059] Third Embodiment Fig. 15 is a diagram illustrating an example of the system configuration of an image analysis system 1 according to a third embodiment. In the example of this figure, the database 20 includes a posture information database 22. The posture information database 22 stores information on various postures that a person can assume, as shown in Fig. 16, for example. Fig. 16 is a diagram illustrating an example of information stored in the posture information database. The posture information database 22 illustrated in this figure stores multiple pieces of posture information including posture images, posture feature amounts (skeletal information), and posture names.
[0060] The image analysis device 10 according to this embodiment has the same functional configuration (e.g., FIG. 1) as the first and second embodiments, except for the points described below.
[0061] The attribute information acquisition unit 110 according to this embodiment further identifies the posture or behavior of each person using skeletal information of each person appearing in the image to be processed. For example, the attribute information acquisition unit 110 compares skeletal information extractable from an image region of a certain target person with skeletal information of various postures registered in the posture information database 22 (hereinafter also referred to as "registered skeletal information"). Based on the comparison result, the attribute information acquisition unit 110 identifies the registered skeletal information with the highest similarity from among the multiple registered skeletal information. Then, the attribute information acquisition unit 110 acquires the name of the posture or behavior (e.g., "handing over" or "sitting") corresponding to the identified registered skeletal information as information indicating the posture or behavior of the target person. The attribute information acquisition unit 110 performs this process for each image to identify the posture or behavior of each person appearing in each image. Then, the attribute information acquisition unit 110 adds information indicating the identified posture or behavior of each person appearing in each image to the attribute information of that person. Furthermore, in this embodiment, the associated person identification unit 130 is configured to be able to narrow down the associated people for a specified person based on the postures and actions of the people included in the attribute information of each person.
[0062] The operation of the image analyzing device 10 according to this embodiment will be described with reference to Fig. 17 to Fig. 19. Fig. 17 to Fig. 19 are diagrams for explaining the operation of the image analyzing device 10 (the associated person identification unit 130 and the display output unit 140) according to the third embodiment.
[0063] First, the associated person identification unit 130 accepts an input specifying a specific posture or behavior. In this case, the display output unit 140 displays a screen, such as the one shown in FIG. 17 , to accept the input specifying the posture or behavior. The screen shown in FIG. 17 has an element E in the upper right portion for setting and changing criteria for identifying an associated person. Unlike the screen shown in the example described above, element E in this figure is configured to be able to accept an input for changing the criteria related to "posture / behavior." The user can, for example, enter an input to change the criteria related to posture or behavior on element E via an input device connected to the input / output interface 1050.
[0064] Here, as illustrated in FIG. 18 , it is assumed that a specific posture (behavior) of "handing over (some object)" is specified by the user. In this case, the associated person identification unit 130 uses attribute information of the specified person acquired from each image to identify an image in which the specified person is depicted in the "handing over" posture. For example, the associated person identification unit 130 can identify an image such as that shown in FIG. 19 . In the image illustrated in FIG. 19 , the specified person (person H) is depicted in the "handing over" posture. Furthermore, in the image illustrated in FIG. 19 , "person C" is depicted as another person. The associated person identification unit 130 narrows down the related persons to the other person ("person C" in the example of FIG. 19 ) who appears together in the identified image. As a result, person F is excluded from the related persons. In accordance with this result, the display output unit 140 updates the screen display as shown in FIG. 18 . 17, only the person relationship information (line L1) indicating the relationship between person H (designated person) and person C (related person) remains on the screen of FIG. 18.
[0065] The new indicator of "person's posture" makes it possible to accurately identify people who are likely to be related to the specified person.
[0066] In this embodiment, the associated person identification unit 130 may further narrow down the associated persons in more detail using a criterion related to "distance," as described in the modified example of the second embodiment.
[0067] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.
[0068] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of steps executed in each embodiment is not limited to the order shown. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of content. Furthermore, the above-described embodiments can be combined as long as the content is not contradictory.
[0069] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. 1. An image analyzing device comprising: attribute information acquisition means for processing a plurality of images to acquire attribute information indicating the attributes of each person appearing in each image; designation input acceptance means for accepting a person designation input indicating a designated person; related person identification means for identifying a related person who is a person related to the designated person using the attribute information acquired for each person; and display output means for outputting a screen including personal relationship information indicating the relationship between the designated person and the related person, and organizational information regarding organizations to which the designated person and the related person belong, respectively. 2. The image analyzing device described in 1., wherein the display output means includes, on the screen as the organizational information, a first organizational chart of the organization to which the designated person belongs and a second organizational chart of the organization to which the related person belongs, and generates the personal relationship information using the first organizational chart and the second organizational chart. 3. The image analyzing device according to any one of 2. to 5., wherein the display output means generates, as the person relationship information, a line that associates the designated person with the related person between the first organizational chart and the second organizational chart. 4. The image analyzing device according to 2., wherein the display output means generates, as the person relationship information, a common mark that associates the designated person with the related person between the first organizational chart and the second organizational chart. 5. The image analyzing device according to 2., wherein the display output means generates the person relationship information by setting a first portion corresponding to the designated person in the first organizational chart and a second portion corresponding to the related person in the second organizational chart to a common display mode. 6. The image analyzing device according to any one of 2. to 5., wherein the display output means displays one or more images in which the designated person indicated by the selected person relationship information and the related person appear together, in response to receiving an input selecting a portion corresponding to the person relationship information on the screen.7. The image analyzing device described in any one of 1. to 6., wherein the associated person identification means calculates, for each person, at least one of the time spent with the designated person and the number of times the person appears with the designated person based on the attribute information of the individual person acquired from each image, and identifies the associated person for the designated person based on a result of comparing the at least one of the time and the number of times calculated for each person with a predetermined criterion for at least one of the time and the number of times. 8. The image analyzing device described in 7., wherein, in response to receiving a criterion change input that changes the criterion, the associated person identification means updates the identification result of the associated person for the designated person based on the new criterion changed by the criterion change input. 9. The image analyzing device described in any one of 1. to 8., wherein the attributes of the person include attributes related to the physical features of the person. 10. 11. An image analysis method comprising: at least one computer processing a plurality of images to acquire attribute information indicating the attributes of each person appearing in each image; accepting a person designation input indicating a designated person; identifying an associated person who is a person related to the designated person using the attribute information acquired for each person; and outputting a screen including personal relationship information indicating the relationship between the designated person and the associated person and organizational information regarding organizations to which the designated person and the associated person belong, respectively. 11. The image analysis method described in 10., including: the at least one computer including, on the screen as the organizational information, a first organizational chart of the organization to which the designated person belongs and a second organizational chart of the organization to which the associated person belongs, and generating the personal relationship information using the first organizational chart and the second organizational chart. 12. The image analysis method described in 11., including: the at least one computer generating, as the personal relationship information, a line that associates the designated person with the associated person between the first organizational chart and the second organizational chart. 13. 12. The image analysis method according to 11., further comprising: generating, as the person relationship information, a common mark that associates the designated person with the related person between the first organizational chart and the second organizational chart, by the at least one computer.14. The image analysis method according to 11., including the at least one computer generating the person relationship information by setting a first portion in the first organizational chart corresponding to the designated person and a second portion in the second organizational chart corresponding to the related person in a common display mode. 15. The image analysis method according to any one of 11. to 14., including the at least one computer displaying one or more images showing the designated person indicated by the selected person relationship information and the related person together, in response to receiving an input selecting a portion corresponding to the person relationship information on the screen. 16. The image analysis method according to any one of 10. to 15., including the at least one computer calculating, for each person, at least one of the time spent with the designated person and the number of times the designated person has appeared together, based on the attribute information of each person acquired from each image, and identifying the related person for the designated person based on a result of comparing the time and / or the number of times calculated for each person with a predetermined standard for at least one of the time and the number of times. 17. The image analysis method according to 16., including the at least one computer updating the identification result of the related person for the designated person based on the new criteria changed by the criteria change input in response to receiving a criteria change input that changes the criteria. 18. The image analysis method according to any one of 10. to 17., wherein the attributes of the person include attributes related to the physical features of the person. 19. A program causing at least one computer to function as: attribute information acquisition means for acquiring attribute information indicating the attributes of each person appearing in each image by processing a plurality of images, designation input accepting means for accepting a person designation input indicating the designated person, related person identification means for identifying a related person who is a person related to the designated person using the attribute information acquired for each person, and display output means for outputting a screen including personal relationship information indicating the relationship between the designated person and the related person, and organization information regarding organizations to which the designated person and the related person each belong.20. The program described in 19., wherein the display output means includes, on the screen as the organizational information, a first organizational chart of the organization to which the designated person belongs and a second organizational chart of the organization to which the related person belongs, and generates the person relationship information using the first organizational chart and the second organizational chart. 21. The program described in 20., wherein the display output means generates, as the person relationship information, a line that associates the designated person with the related person between the first organizational chart and the second organizational chart. 22. The program described in 20., wherein the display output means generates, as the person relationship information, a common mark that associates the designated person with the related person between the first organizational chart and the second organizational chart. 23. The program described in 20., wherein the display output means generates the person relationship information by setting a first portion corresponding to the designated person in the first organizational chart and a second portion corresponding to the related person in the second organizational chart to a common display mode. 24. The program according to any one of 20. to 23., wherein the display output means displays one or more images showing the designated person indicated by the selected personal relationship information and the related person together in response to receiving an input selecting a portion corresponding to the personal relationship information on the screen. 25. The program according to any one of 19. to 24., wherein the related person identification means calculates, for each person, at least one of the time spent with the designated person and the number of times the person has appeared with the designated person based on the attribute information of the individual person acquired from each image, and identifies the related person for the designated person based on a result of comparing the calculated time and / or number of times with a predetermined criterion for at least one of the time and number of times. 26. The program according to 25., wherein, in response to receiving a criterion change input to change the criterion, the related person identification means updates the identification result of the related person for the designated person based on the new criterion changed by the criterion change input. 27. The attributes of the person include attributes related to the person's physical characteristics. 1. A program according to any one of the preceding claims.
[0070] This application claims priority based on Japanese Patent Application No. 2023-075670, filed May 1, 2023, the disclosure of which is incorporated herein in its entirety by reference.
[0071] REFERENCE SIGNS LIST 1 Image analysis system 10 Image analysis device 1000 Computer 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input / output interface 1060 Network interface 110 Attribute information acquisition unit 120 Designation input reception unit 130 Related person identification unit 140 Display output unit 20 Database 22 Posture information database 30 Display 40 Network
Claims
1. An image analysis device comprising: an attribute information acquisition means for processing a plurality of images to acquire attribute information indicating the attributes of each person appearing in each image; a designated input receiving means for receiving a person designation input indicating a designated person; a related person identification means for identifying a related person who is a person related to the designated person using the attribute information acquired for each of the people; and a display output means for outputting a screen including personal relationship information indicating the relationship between the designated person and the related person, and organizational information regarding organizations to which the designated person and the related person each belong.
2. The image analysis device of claim 1, wherein the display output means includes on the screen a first organizational chart of the organization to which the specified person belongs and a second organizational chart of the organization to which the related person belongs as the organizational information, and generates the person relationship information using the first organizational chart and the second organizational chart.
3. The image analysis device according to claim 2, wherein the display output means generates a line that corresponds the specified person to the related person between the first organizational chart and the second organizational chart as the person relationship information.
4. The image analysis device according to claim 2, wherein the display output means generates a common mark that corresponds the specified person and the related person between the first organizational chart and the second organizational chart as the person relationship information.
5. The image analysis device of claim 2, wherein the display output means generates the person relationship information by setting a first portion in the first organizational chart corresponding to the specified person and a second portion in the second organizational chart corresponding to the related person in a common display mode.
6. The image analysis device according to any one of claims 2 to 5, wherein the display output means displays one or more images showing both the designated person and the related person indicated by the selected personal relationship information in response to receiving an input to select a portion on the screen corresponding to the personal relationship information.
7. The image analysis device described in any one of claims 1 to 5, wherein the related person identification means calculates, for each person, at least one of the time spent with the designated person and the number of times the person appears with the designated person based on the attribute information of each person obtained from each image, and identifies the related person for the designated person based on the result of comparing the time and / or the number of times calculated for each person with a predetermined standard for at least one of the time and the number of times.
8. The image analysis device according to claim 7, wherein the associated person identification means, in response to receiving a criterion change input that changes the criteria, updates the identification results of the associated person for the specified person based on the new criteria changed by the criterion change input.
9. The image analysis device according to any one of claims 1 to 5, wherein the attributes of the person include attributes related to physical characteristics of the person.
10. An image analysis method comprising: at least one computer processing a plurality of images to obtain attribute information indicating the attributes of each person appearing in each image; accepting a person designation input indicating a designated person; identifying related persons who are persons related to the designated person using the attribute information obtained for each person; and outputting a screen including personal relationship information indicating the relationship between the designated person and the related persons, and organizational information regarding organizations to which the designated person and the related persons each belong.
11. The image analysis method according to claim 10, further comprising the step of: including on the screen, as the organizational information, a first organizational chart of the organization to which the specified person belongs and a second organizational chart of the organization to which the related person belongs; and generating the person relationship information using the first organizational chart and the second organizational chart.
12. The image analysis method according to claim 11, further comprising: generating, as the person relationship information, a line that corresponds the specified person to the related person between the first organizational chart and the second organizational chart, by the at least one computer.
13. The image analysis method according to claim 11, further comprising: generating, as the person relationship information, a common mark that corresponds the specified person to the related person between the first organizational chart and the second organizational chart, by the at least one computer.
14. The image analysis method according to claim 11, further comprising: generating the person relationship information by setting a first portion in the first organizational chart corresponding to the specified person and a second portion in the second organizational chart corresponding to the related person in a common display mode.
15. An image analysis method according to any one of claims 11 to 14, comprising the step of: displaying, in response to receiving an input to select on the screen a portion corresponding to the person relationship information, one or more images in which the designated person indicated by the selected person relationship information and the related person are both shown.
16. The image analysis method according to any one of claims 10 to 15, comprising the step of: calculating, for each person, at least one of the time spent with the designated person and the number of times the person has appeared with the designated person based on the attribute information of each person obtained from each image; and identifying the related persons for the designated person based on the result of comparing the time and / or the number of times calculated for each person with a predetermined standard for at least one of the time and the number of times.
17. The image analysis method according to claim 16, further comprising: updating, in response to receiving a criterion change input that changes the criteria, the at least one computer updates the identification results of the related persons for the specified person based on the new criteria changed by the criterion change input.
18. The image analysis method according to any one of claims 10 to 17, wherein the attributes of the person include attributes related to physical characteristics of the person.
19. A computer-readable recording medium having recorded thereon a program that, when executed by at least one computer, causes the computer to perform operations including: processing a plurality of images to obtain attribute information indicating the attributes of each person appearing in each image; accepting a person designation input indicating a designated person; identifying associated persons who are persons related to the designated person using the attribute information obtained for each person; and outputting a screen including personal relationship information indicating the relationship between the designated person and the associated person, and organizational information regarding the organizations to which the designated person and the associated person each belong.
Citation Information
Patent Citations
Information processing system, administrative server and information processing method
JP2013045160A
Human relationship estimation system
JP2013182560A
Personal correlation chart providing device and personal correlation chart providing system
JP2018200663A
METHOD, IDENTIFICATION DEVICE AND IDENTIFICATION PROGRAM FOR DISCOVERING MULTI-LAYER POTENTIAL MATE - Patent application
JP2021536182A
