Information processing system, information processing device, terminal, processing method, and program

Abstract

To provide an image processing device capable of monitoring a suspicious person even when the suspicious person is not known.SOLUTION: An image processing system comprises: a detection unit which with respect to a person appearing in a video captured from a camera selected based on an input to an input device, detects a gesture of the person; and an estimation unit which estimates a predetermined person candidate based on a result of the detection by the detection unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, a terminal, a processing method, and a program. [Background technology]

[0002] The installation of security cameras is increasing in apartment buildings, houses, urban areas, and more. Patent Document 1 describes a related technique for monitoring suspicious persons. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-191620 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Patent Document 1 is premised on the assumption that suspicious persons are known in advance, and cannot monitor suspicious persons if they are not known.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an information processing system, an information processing device, a terminal, a processing method, and a program that can solve the above-mentioned problems. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, according to one aspect of the present disclosure, an image processing device includes an appearance state determination unit that determines the appearance state of a person in a captured image, and a specified person candidate estimation unit that estimates a specified person candidate based on the appearance state.

[0007] According to one aspect of the present disclosure, the image processing system includes a camera and an image processing device, the camera captures an image of a specified area, and the image processing device includes an appearance state determination unit that determines an appearance state of a person in the captured image, and a specified person candidate estimation unit that estimates a specified person candidate based on the appearance state.

[0008] According to another aspect of the present disclosure, an image processing method determines an appearance state of a person in a captured image, and estimates a predetermined person candidate based on the appearance state.

[0009] In addition, according to one aspect of the present disclosure, the program causes a computer to function as an appearance state determination means for determining the appearance state of a person in a captured image, and a specified person candidate estimation means for estimating a specified person candidate based on the appearance state. Effect of the Invention

[0010] By processing the image processing system of the present invention, it is possible to identify person candidates based on specified conditions, such as suspicious persons. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing a minimum configuration of an image processing device according to an embodiment of the present invention. [Diagram 2] 1 is a diagram showing a configuration of an image processing system according to a first embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram showing a configuration of a center system according to the present embodiment. [Figure 4] 1 is a diagram showing a configuration of an input device according to an embodiment of the present invention; [Diagram 5] FIG. 2 is a diagram illustrating a configuration of a first edge device according to the present embodiment. [Figure 6] FIG. 2 is a diagram illustrating a configuration of a second edge device according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing the configuration of a third edge device according to the present embodiment. [Figure 8] FIG. 4 is a diagram showing a first data table according to the present embodiment. [Figure 9] FIG. 11 is a diagram showing a second data table according to the present embodiment. [Figure 10] FIG. 13 is a diagram showing a third data table according to the embodiment. [Figure 11] FIG. 13 is a diagram showing a general data table according to the present embodiment. [Figure 12] FIG. 1 is a first diagram showing a processing flow according to the present embodiment. [Figure 13] FIG. 2 is a second diagram showing the processing flow according to the present embodiment. [Figure 14] 1 is a diagram showing a configuration of an input device according to a first embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] FIG. 1 is a diagram showing a minimum configuration of an image processing device 100 according to an embodiment of the present invention. As shown in FIG. 1, an image processing device 100 of the present invention includes at least an appearance state determination unit 101 and a predetermined person candidate estimation unit . The appearance state determination unit 101 determines the appearance state of a person in a captured image. The appearance state of a person is, for example, a result indicating the appearance of a person, such as a person appearing many times in the image, a person appearing for a long time, or a person behaving suspiciously, such as frequently looking around the surroundings. The predetermined person candidate estimation unit 102 estimates a predetermined person candidate based on the appearance state. The predetermined person candidate is, for example, a shoplifter, a suspicious purchaser, a stalker, or the like.

[0013] First Embodiment FIG. 2 is a diagram showing the configuration of an image processing system 1 according to the first embodiment of the present invention. First, an image processing system 1 including an image processing device 100 according to a first embodiment of the present invention will be described. As shown in FIG. 2, the image processing system 1 according to this embodiment includes a center system 10, an input device 20, a first edge device 30a (information processing device), a second edge device 30b (information processing device), a third edge device 30c (information processing device), a first camera device 40a, a second camera device 40b, and a third camera device 40c. In the following description, the first edge device 30a, the second edge device 30b, the third edge device 30c, ..., the nth edge device 30n, ... will be collectively referred to as edge device 30. In addition, the first camera device 40a, the second camera device 40b, the third camera device 40c, ..., the nth camera device 40n, ... will be collectively referred to as camera device 40.

[0014] The input device 20 is, for example, a smartphone, a mobile phone, a personal computer, etc. The user performs a camera selection information input operation to input camera selection information for selecting a camera that is to extract a person's feature amount, which is a person's feature amount. Note that the person feature amount is information for identifying a person, such as face, height, sex, clothing, belongings, etc.

[0015] The user performs a start instruction input operation to input a start instruction to start extraction of person feature amounts. The input device 20 transmits camera selection information to the center system 10 based on the start instruction input operation by the user.

[0016] The center system 10 receives the camera selection information from the input device 20. The center system 10 identifies an edge device based on the input camera selection information. The center system 10 transmits camera selection information and a feature extraction start instruction to start extraction of person features to each of the identified first edge device 30a, second edge device 30b, and third edge device 30c.

[0017] When the first edge device 30a, the second edge device 30b, and the third edge device 30c receive the camera selection information and the person feature extraction start instruction from the center system 10, they acquire a camera image from the camera device 40 having the camera indicated by the camera selection information. Then, the first edge device 30a, the second edge device 30b, and the third edge device 30c each extract a person feature from the acquired camera image. The first edge device 30a, the second edge device 30b, and the third edge device 30c each generate an index for the extracted person feature. The index is information including, for example, an input device ID (IDentifier) ​​which is an identifier of the input device 20 that transmitted the camera selection information, a person feature ID which is an identifier of the person feature assigned to each person feature, and the like. The first edge device 30a, the second edge device 30b, and the third edge device 30c each transmit the generated index together with the extracted person feature to the center system 10. In addition, the first edge device 30a, the second edge device 30b, and the third edge device 30c each record the generated index together with the extracted person feature in a storage unit. In addition, the first edge device 30a, the second edge device 30b, and the third edge device 30c each assign the generated index to the camera image and record the camera image in a storage unit.

[0018] When the center system 10 receives the person features and index along with the camera images from each of the first edge device 30a, the second edge device 30b, and the third edge device 30c, it associates the received camera images, person features, and index and records them in a memory unit.

[0019] The user also performs a suspicious person extraction start instruction input operation to input a suspicious person extraction start instruction to instruct the start of suspicious person extraction. The input device 20 transmits a suspicious person extraction start instruction to the center system 10 based on the suspicious person extraction start instruction from the user.

[0020] The center system 10 inputs candidate person determination conditions for determining a candidate person such as a suspicious person, and records the input in the storage unit. The candidate person determination conditions are determination conditions for determining an appearance state, which is a result indicating the appearance of a person, such as a large number of times a person appears in the video or a person appears for a long time. When the center system 10 inputs a suspicious person extraction start instruction that instructs the start of extraction of a suspicious person, the center system 10 identifies the suspicious person based on the camera video, person feature amount, index, and appearance state stored in the storage unit. The center system 10 records the suspicious person identification result for the identified suspicious person in the storage unit.

[0021] The user performs an end instruction input operation to input an end instruction to end the extraction of personal feature amounts. The input device 20 transmits the end instruction to the center system 10 based on the end instruction input operation by the user.

[0022] When the center system 10 receives the end instruction from the input device 20, it transmits a feature extraction end instruction indicating the end of extraction of person features to each of the first edge device 30a, the second edge device 30b, and the third edge device 30c.

[0023] When the first edge device 30a, the second edge device 30b, and the third edge device 30c receive the feature extraction end instruction from the center system 10, they each end the extraction of human feature amounts from the camera video.

[0024] As described above, in the image processing system 1 according to the embodiment of the present invention, the camera device 40 captures an image in an area selected by a user. Each of the edge devices 30 extracts person features from the image captured by the camera device 40. The center system 10 identifies, as a suspicious person, a person who satisfies an appearance state determined using a person candidate determination condition from among the people having the person features extracted by the camera device 40.

[0025] FIG. 3 is a diagram showing the configuration of the center system 10 according to this embodiment. Next, the configuration of the center system 10 according to this embodiment will be described. 3, the center system 10 according to this embodiment includes an image processing device 100. The image processing device 100 includes an appearance state determination unit 101, a predetermined person candidate estimation unit 102, a communication unit 103, a control unit 104, a camera selection unit 105, a storage unit 106, and a condition input unit 107.

[0026] The appearance state determination unit 101 determines the appearance state of a person in a captured video based on the person candidate determination conditions. The predetermined person candidate estimation unit 102 estimates a predetermined person candidate based on the appearance state of a person in a captured image. For example, the predetermined person candidate estimation unit 102 identifies a person in a camera image having a person feature amount satisfying the appearance state as a suspicious person based on the camera image, the person feature amount, the index, and the appearance state received by the center system 10 from each of the edge devices 30. The communication unit 103 transmits and receives information to and from each of the input device 20 and the edge device 30 .

[0027] The control unit 104 performs various controls necessary for the processing performed by the center system 10. For example, the control unit 104 controls the transmission and reception of information performed by the communication unit 103. Note that in the following description, even if it is not specifically stated that the control unit 104 controls the transmission and reception of information performed by the communication unit 103, the control unit 104 controls the transmission and reception of information performed by the communication unit 103.

[0028] The camera selection unit 105 designates a camera for capturing the appearance state of a person in the captured video determined by the appearance state determination unit 101. The storage unit 106 stores information necessary for various processes performed by the center system 10. For example, the storage unit 106 stores the camera video, the person feature amount, and the index received by the center system 10 from each edge device 30 in association with each other. The condition input unit 107 acquires the person candidate determination conditions, and records the acquired person candidate determination conditions in the storage unit 106 .

[0029] FIG. 4 is a diagram showing a configuration of the input device 20 according to this embodiment. Next, the configuration of the input device 20 according to this embodiment will be described. The input device 20 according to this embodiment includes a communication unit 201, an information input unit 202, and a display control unit 203, as shown in FIG.

[0030] The communication unit 201 transmits and receives information to and from the center system 10 . The information input unit 202 detects an input operation by the user and outputs a signal based on the detected input operation. For example, when the information input unit 202 detects a camera selection information input operation by the user, it transmits the camera selection information to the center system 10. Also, for example, when the information input unit 202 detects a start instruction input operation by the user, it transmits a start instruction to the center system 10. Also, for example, when the information input unit 202 detects an end instruction input operation by the user, it transmits an end instruction to the center system 10.

[0031] The display control unit 203 controls the display on a display screen (not shown). For example, the display control unit 203 controls the display on the display screen of information that the input device 20 transmits to the center system 10 and information that the input device 20 receives from the center system 10.

[0032] Next, the configuration of the edge device 30 according to this embodiment will be described. FIG. 5 is a diagram showing the configuration of the first edge device 30a according to this embodiment. First, the configuration of the first edge device 30a according to this embodiment will be described. As shown in FIG. 5, the first edge device 30a according to this embodiment includes a first communication unit 301a, a first control unit 302a, a first camera control unit 303a, a first memory unit 304a, a first feature extraction unit 305a, a first video receiving unit 306a, and a first camera interface 307a.

[0033] The first communication unit 301a transmits and receives information to and from the center system 10 and the first camera device 40a. The first control unit 302a performs various controls necessary for the processing performed by the first edge device 30a. For example, the first control unit 302a controls the transmission and reception of information performed by the first communication unit 301a. Note that in the following description, even if it is not specifically described that the first control unit 302a controls the transmission and reception of information performed by the first communication unit 301a, the first control unit 302a controls the transmission and reception of information performed by the first communication unit 301a.

[0034] The first camera control unit 303a transmits a shooting control signal for controlling shooting to the first camera device 40a via the first camera interface 307a. The first storage unit 304a stores various information necessary for the processing performed by the first edge device 30a.

[0035] The first feature extraction unit 305a extracts person features captured by the camera included in the camera device 40 specified by the camera selection unit 105. Specifically, the first video receiving unit 306a performs image analysis on the video received from the first camera device 40a to extract the person features.

[0036] The first video receiving unit 306a receives the video captured by the first camera device 40a. The first camera interface 307a is an interface provided in the first edge device 30a for connecting the first edge device 30a and the first camera device 40a.

[0037] FIG. 6 is a diagram showing the configuration of the second edge device 30b according to this embodiment. Next, the configuration of the second edge device 30b according to this embodiment will be described. As shown in FIG. 6, the second edge device 30b according to this embodiment includes a second communication unit 301b, a second control unit 302b, a second camera control unit 303b, a second memory unit 304b, a second feature extraction unit 305b, a second video receiving unit 306b, and a second camera interface 307b.

[0038] The second communication unit 301b transmits and receives information to and from the center system 10 and the second camera device 40b. The second control unit 302b performs various controls necessary for the processing performed by the second edge device 30b. For example, the second control unit 302b controls the transmission and reception of information performed by the second communication unit 301b. Note that in the following description, even if it is not specifically described that the second control unit 302b controls the transmission and reception of information performed by the second communication unit 301b, the second control unit 302b controls the transmission and reception of information performed by the second communication unit 301b.

[0039] The second camera control unit 303b transmits a shooting control signal for controlling shooting to the second camera device 40b via the second camera interface 307b. The second storage unit 304b stores various information necessary for the processing performed by the second edge device 30b.

[0040] The second feature extraction unit 305b extracts person features captured by the camera included in the camera device 40 specified by the camera selection unit 105. Specifically, the second video receiving unit 306b performs image analysis on the video received from the second camera device 40b to extract the person features.

[0041] The second video receiving unit 306b receives the video captured by the second camera device 40b. The second camera interface 307b is an interface provided in the second edge device 30b for connecting the second edge device 30b and the second camera device 40b.

[0042] FIG. 7 is a diagram showing the configuration of the third edge device 30c according to this embodiment. Next, the configuration of the third edge device 30c according to this embodiment will be described. As shown in FIG. 7, the third edge device 30c according to this embodiment includes a third communication unit 301c, a third control unit 302c, a third camera control unit 303c, a third memory unit 304c, a third feature extraction unit 305c, a third video receiving unit 306c, and a third camera interface 307c.

[0043] The third communication unit 301c transmits and receives information to and from the center system 10 and the third camera device 40c. The third control unit 302c performs various controls necessary for the processing performed by the third edge device 30c. For example, the third control unit 302c controls the transmission and reception of information performed by the third communication unit 301c. Note that in the following description, even if it is not specifically described that the third control unit 302c controls the transmission and reception of information performed by the third communication unit 301c, the third control unit 302c controls the transmission and reception of information performed by the third communication unit 301c.

[0044] The third camera control unit 303c transmits an image capture control signal for controlling image capture to the third camera device 40c via the third camera interface 307c. The third storage unit 304c stores various information necessary for the processing performed by the third edge device 30c.

[0045] The third feature extraction unit 305c extracts human features captured by the camera included in the camera device 40 specified by the camera selection unit 105. For example, the third video receiving unit 306c performs image analysis on the video received from the third camera device 40c to extract the face, height, sex, clothing, belongings, and the like.

[0046] The third video receiving unit 306c receives the video captured by the third camera device 40c. The third camera interface 307c is an interface provided in the third edge device 30c for connecting the third edge device 30c to the third camera device 40c.

[0047] Next, the configuration of the camera device 40 according to this embodiment will be described. Each of the camera devices 40 includes one or more cameras. In this embodiment, the first camera device 40a includes a camera 401a, a camera 402a, and a camera 403a. The second camera device 40b includes a camera 401b, a camera 402b, and a camera 403b. The third camera device 40c includes a camera 401c and a camera 402c.

[0048] Next, the data tables stored in each of the first edge device 30a, the second edge device 30b, and the third edge device 30c will be described. FIG. 8 is a diagram showing the first data table TBL1 according to this embodiment. The first storage unit 304a stores the person feature extracted by the first feature extraction unit 305a in the first data table TBL1. For example, as shown in FIG. 8, the first storage unit 304a stores the person feature extracted by the first feature extraction unit 305a from the video captured by each of the cameras 401a, 402a, and 403a in association with the date and time and an index for each camera. The capture date and time in the first data table TBL1 shown in FIG. 8 is the date and time when each of the cameras 401a, 402a, and 403a started capturing the video portion from which the first feature extraction unit 305a is extracting the person feature. The release date and time in the first data table TBL1 shown in FIG. 8 is the date and time when each of the cameras 401a, 402a, and 403a finished capturing the video portion from which the first feature extraction unit 305a is extracting the person feature. The index is information including, for example, an input device ID that is an identifier of the input device 20 that has transmitted the camera selection information, a person feature ID that is an identifier of a person feature assigned to each person feature, and the like.

[0049] FIG. 9 is a diagram showing the second data table TBL2 according to this embodiment. The second storage unit 304b stores the person features extracted by the second feature extraction unit 305b in a second data table TBL2. For example, as shown in Fig. 9, the second storage unit 304b stores the person features extracted by the second feature extraction unit 305b from the videos captured by the cameras 401b, 402b, and 403b in association with the date and time and the index for each camera.

[0050] FIG. 10 is a diagram showing the third data table TBL3 according to this embodiment. The third storage unit 304c stores the person features extracted by the third feature extraction unit 305c in a third data table TBL3. For example, as shown in Fig. 10, the third storage unit 304c stores the person features extracted by the third feature extraction unit 305c from the videos captured by the camera 401c and the camera 402c in association with the date and time and the index for each camera.

[0051] Next, the general data table STBL stored in the center system 10 will be described. FIG. 11 is a diagram showing the general data table STBL according to this embodiment. The storage unit 106 (person feature amount storage unit) stores the person feature amounts extracted by the first feature amount extraction unit 305a, the second feature amount extraction unit 305b, and the third feature amount extraction unit 305c in a comprehensive data table STBL. For example, the storage unit 106 stores the comprehensive data table STBL as shown in FIG. 11, in which the first data table TBL1, the second data table TBL2, and the third data table TBL3 are combined into one data table.

[0052] 12 and 13 are diagrams showing the processing flow of the image processing system 1 according to this embodiment. Next, the processing of the image processing system 1 according to this embodiment will be described. Here, the process will be described in which the image processing system 1 monitors suspicious persons in multiple stores of the same chain. The first edge device 30a and the first camera device 40a are assumed to be located in the first store A. The second edge device 30b and the second camera device 40b are assumed to be located in the second store B. The third edge device 30c and the third camera device 40c are assumed to be located in the third store C. In the following description, an example will be described in which the user is an employee of the first store A and the person feature amount is a face. Although the first edge device 30a, the second edge device 30b, and the third edge device 30c each perform information processing independently, they operate in a similar manner, and therefore are referred to as the edge device 30 in the processing flows shown in FIGS.

[0053] First, the human feature extraction process performed by the image processing system 1 according to this embodiment will be described with reference to the processing flow of FIG. The user performs a camera selection information input operation on the input device 20 to select cameras installed in a plurality of stores of the same chain from which faces are to be extracted. The information input unit 202 inputs camera selection information according to a camera selection information input operation by the user (step S1). The camera selection information indicates that all of the cameras installed in the first store A, the second store B, and the third store C are to be selected. The cameras installed in the first store A are camera 401a, camera 402a, and camera 403a. The cameras installed in the second store B are camera 401b, camera 402b, and camera 403b. The cameras installed in the third store C are camera 401c and camera 402c. Then, the information input unit 202 determines whether to start face extraction (step S2). If the information input unit 202 does not detect a start instruction input operation for inputting a start instruction to start face extraction, the information input unit 202 returns to the process of step S2.

[0054] The user performs a start instruction input operation on the input device 20 to input a start instruction to start face extraction. The information input unit 202 detects a start instruction input operation by the user (step S2, YES), and transmits camera selection information to the center system 10 (step S3).

[0055] The camera selection unit 105 determines whether or not to receive camera selection information (step S4). If the camera selection unit 105 does not receive camera selection information, the process returns to step S4. When the camera selection unit 105 receives the camera selection information from the input device 20 (step S4, YES), it specifies an edge device based on the received camera selection information (step S5). Specifically, the camera selection unit 105 specifies the first edge device 30a corresponding to the camera 401a, the camera 402a, and the camera 403a included in the camera selection information, the second edge device 30b corresponding to the camera 401b, the camera 402b, and the camera 403b, and the third edge device 30c corresponding to the camera 401c and the camera 402c. For example, the camera selection unit 105 sequentially compares the cameras in the data table showing the correspondence between the cameras and the edge devices with the cameras included in the received camera selection information, and specifies the matching cameras. The camera selection unit 105 specifies the edge device corresponding to the matching camera as the selected edge device.

[0056] The camera selection unit 105 transmits camera selection information and a feature extraction start instruction to start extraction of human features to each of the identified first edge device 30a, second edge device 30b, and third edge device 30c (step S6).

[0057] The first video receiving unit 306a determines whether or not a person feature extraction start instruction has been received (step S7), and determines whether or not to start person feature extraction based on the determination result.

[0058] When the first video receiving unit 306a determines that it has not received the instruction to start extracting person features (step S7, NO), it determines not to start extracting person features. In this case, the first video receiving unit 306a receives camera videos from the cameras 401a, 402a, and 403a (step S8), and records the received camera videos in the first storage unit 304a (step S9).

[0059] Furthermore, when the first video receiving unit 306a determines that it has received a person feature extraction start instruction (step S7, YES), it determines to start person feature extraction. In this case, the first video receiving unit 306a receives camera videos from the cameras 401a, 402a, and 403a indicated by the camera selection information (step S10). The first feature extraction unit 305a extracts a face from each camera video received by the first video receiving unit 306a (step S11). The first feature extraction unit 305a generates an index for the extracted face (step S12). The first feature extraction unit 305a transmits the generated index together with the extracted face to the center system 10 (step S13). The first feature extraction unit 305a also records the generated index together with the facial information of the extracted face in the first data table TBL1 of the first storage unit 304a (step S14). The first feature extraction unit 305a also assigns the generated index to each camera video (step S15) and records each camera video in the first storage unit 304a (step S16).

[0060] Similar to the first video receiving unit 306a, when it is determined that the person feature extraction start instruction has been received (step S7, YES), the second video receiving unit 306b determines to start the person feature extraction. In this case, the second video receiving unit 306b receives camera videos from the cameras 401b, 402b, and 403b indicated by the camera selection information (step S10). The second feature extraction unit 305b extracts a face from each camera video received by the second video receiving unit 306b (step S11). The second feature extraction unit 305b generates an index for the extracted face (step S12). The second feature extraction unit 305b transmits the generated index together with the extracted face to the center system 10 (step S13). The second feature extraction unit 305b also records the generated index together with the facial information of the extracted face in the second data table TBL2 of the second storage unit 304b (step S14). The second feature extraction unit 305b also assigns the generated index to each camera video (step S15) and records each camera video in the second storage unit 304b (step S16).

[0061] Similar to the first video receiving unit 306a, when it is determined that the person feature extraction start instruction has been received (step S7, YES), the third video receiving unit 306c determines to start the person feature extraction. In this case, the third video receiving unit 306c receives camera videos from the camera 401c and the camera 402c indicated by the camera selection information (step S10). The third feature extraction unit 305c extracts a face from each camera video received by the third video receiving unit 306c (step S11). The third feature extraction unit 305c generates an index for the extracted face (step S12). The third feature extraction unit 305c transmits the generated index together with the extracted face to the center system 10 (step S13). The third feature extraction unit 305c also records the generated index together with the facial information of the extracted face in the third data table TBL3 of the third storage unit 304c (step S14). The third feature extraction unit 305c also assigns the generated index to each camera video (step S15) and records each camera video in the third storage unit 304c (step S16).

[0062] When the appearance state determination unit 101 receives face information and an index from each of the first edge device 30a, the second edge device 30b, and the third edge device 30c (step S17), it associates the received face information and index and records them in the general data table STBL of the memory unit 106 (step S18). At this time, the center system 10 does not receive camera images from each of the first edge device 30a, the second edge device 30b, and the third edge device 30c, but receives only face information (person feature amount) and an index, thereby reducing the amount of communication data. However, the center system 10 may receive camera images together with face information (person feature amount) and an index from each of the first edge device 30a, the second edge device 30b, and the third edge device 30c. In addition, the center system 10 may record the camera images received from each of the first edge device 30a, the second edge device 30b, and the third edge device 30c in a storage unit.

[0063] Thereafter, the user performs an end instruction input operation to input an end instruction to end the extraction of personal feature amounts. The information input unit 202 inputs an end instruction based on the end instruction input operation by the user (step S19). The information input unit 202 transmits the end instruction to the center system 10 (step S20).

[0064] The appearance state determining unit 101 constantly determines whether or not to end the extraction of person features (step S21). When the appearance state determination unit 101 does not receive an end instruction from the input device 20 (step S21, NO), the process returns to step S17. In addition, when the appearance state determination unit 101 receives an end instruction from the input device 20 (step S21, YES), it sends a feature extraction end instruction indicating the end of the extraction of person features to each of the first edge device 30a, the second edge device 30b, and the third edge device 30c (step S22).

[0065] The first feature extraction unit 305a determines whether or not a feature extraction end instruction has been received from the center system 10 (step S23), and determines whether or not to end extraction of faces in the camera video based on the result of the determination.

[0066] When the first feature extraction unit 305a determines that the feature extraction end instruction has not been received (step S23, NO), it determines not to end the extraction of the face in the camera video. In this case, the first feature extraction unit 305a returns to the process of step S10.

[0067] Furthermore, when the first video receiving unit 306a determines that it has received the instruction to end the feature extraction (step S23, YES), it determines to end the extraction of the face in the camera video. In this case, the first video receiving unit 306a returns to the process of step S8. The above is the person feature extraction process performed by the image processing system 1 according to this embodiment.

[0068] Next, a process of estimating a candidate for a predetermined person performed by the image processing system 1 according to this embodiment will be described with reference to the process flow of FIG. After the extraction process of the person features, the user who operated the input device 20, or another user who identifies the suspicious person (e.g., a police officer, etc.), performs a suspicious person extraction start instruction input operation to input a suspicious person extraction start instruction to start extraction of the suspicious person, either directly to the center system 10 or via a device installed remotely and connected by communication.

[0069] The storage unit 106 stores in advance the person candidate determination conditions for determining a suspicious person inputted via the condition input unit 107. For example, the person candidate determination conditions for determining a suspicious person are conditions for identifying a person who may shoplift, such as entering and leaving the same store many times on the same day, staying at the same store for a long time on the same day but not purchasing any products, entering and leaving multiple stores of the same chain many times on the same day, staying at multiple stores of the same chain for a long time on the same day but not purchasing any products, and frequently visiting multiple stores of the same chain over multiple days but not purchasing any products. In addition, the person candidate determination conditions for determining a suspicious person are conditions for identifying a person who may make a large purchase, such as the number of times a person appears and the time of a person's appearance, which are determined based on entering and leaving the same store many times on the same day and purchasing products many times, entering and leaving multiple stores of the same chain many times on the same day and purchasing products many times, and frequently visiting each store over multiple days and purchasing products many times.

[0070] When the predetermined person candidate estimation unit 102 receives a suspicious person extraction start command (step S31), it identifies a suspicious person based on the camera image, face information, index, and appearance state stored in the storage unit 106 (step S32). Specifically, the predetermined person candidate estimation unit 102 rearranges data for each face information (feature amount) in the general data table STBL stored in the storage unit 106, identifies the number of appearances of each face information, the appearance insertion time of each face information, and the like, and identifies the person corresponding to the face information as a suspicious person when the person candidate determination condition is satisfied. Note that, when a face is extracted in the camera image of the camera arranged at the position of the cash register, the predetermined person candidate estimation unit 102 determines that the person with that face has purchased a product. Also, when a face is not extracted in the camera image of the camera arranged at the position of the cash register, the predetermined person candidate estimation unit 102 determines that the person with that face has not purchased a product. Furthermore, the predetermined person candidate presumption unit 102 counts the number of appearances of a person with that face according to the number of times the face is extracted from the camera image of the camera placed at the entrance / exit of the store. The predetermined person candidate presumption unit 102 records the camera image and the index, which are the suspicious person identification results, for the identified suspicious person in the storage unit 106 (step S33). The above is the process of estimating a candidate for a predetermined person performed by the image processing system 1 according to this embodiment. When the predetermined person candidate estimation unit 102 identifies a suspicious person, the predetermined person candidate estimation unit 102 may transmit to the input device 20 camera image data showing the appearance of the suspicious person, such as the face, height, sex, clothing, and belongings of the suspicious person, based on the camera image from which the feature amount of the suspicious person has been extracted. In this case, the display control unit 203 causes the center system 10 to display the camera image data showing the appearance of the suspicious person on the display unit, so that the user can confirm the suspicious person. In addition, the predetermined person candidate estimation unit 102 may provide the camera image data showing the appearance of the suspicious person to the Metropolitan Police Department, etc. In this case, it can be useful for crime prevention.

[0071] The image processing system 1 according to the first embodiment of the present invention has been described above. In the image processing device 100 of the above-mentioned image processing system 1, the appearance state determination unit 101 determines the appearance state of a person in a captured image. The predetermined person candidate estimation unit 102 estimates a predetermined person candidate based on the appearance state determined by the appearance state determination unit 101. This makes it possible to identify suspected individuals based on specified criteria, such as suspicious people.

[0072] Second Embodiment Next, an image processing system 1 including an image processing device 100 according to a second embodiment of the present invention will be described. The image processing system 1 according to this embodiment, like the image processing system 1 shown in FIG. 2, includes a center system 10, an input device 20, a first edge device 30a, a second edge device 30b, a third edge device 30c, a first camera device 40a, a second camera device 40b, and a third camera device 40c. The image processing system 1 according to this embodiment differs from the image processing system 1 according to the first embodiment in the input device 20.

[0073] FIG. 14 is a diagram showing the configuration of an input device 20 according to this embodiment. Next, the configuration of the input device 20 according to this embodiment will be described. As shown in FIG. 14, the input device 20 according to this embodiment further includes a position identification unit 204 and a camera selection information generation unit 205 in addition to the communication unit 201, information input unit 202, and display control unit 203 shown in FIG. 4.

[0074] The position identification unit 204 identifies the position of the input device 20. The communication unit 201 transmits and receives information to and from the center system . The camera selection information generating unit 205 generates camera selection information for identifying a camera that captures the appearance of a person in the captured video, based on the position of the input device 20 identified by the position identifying unit 204. For example, the camera selection information generating unit 205 generates camera selection information for selecting a camera that meets a predetermined condition, such as a camera installed within 1 kilometer from the position of the input device 20 identified by the position identifying unit 204, or a camera installed closest to the position of the input device 20.

[0075] Next, the processing of the image processing system 1 according to this embodiment will be described. Here, a process will be described in which the image processing system 1 monitors a stalker. In this embodiment, a person who is a victim of a stalker is the user. The processing flow of the image processing system 1 according to this embodiment is the same as that of the image processing system 1 according to the first embodiment shown in FIGS. 12 and 13, except for the processing in steps S1 and S3.

[0076] The information input unit 202 does not perform the process of inputting the camera selection information in step S1, and determines whether or not to start face extraction in step S2. When the information input unit 202 does not detect a start instruction input operation for inputting a start instruction for starting face extraction, the information input unit 202 returns to the process of step S2. When the information input unit 202 detects a start instruction input operation, instead of the process of step S3, the position identification unit 204 identifies the position of the input device 20 at a predetermined time interval while the user carrying the input device 20 is moving. For example, the position identification unit 204 identifies the position of the input device 20 using a function of a GPS (Global Positioning System). The position identification unit 204 outputs the identified position of the input device 20 to the camera selection information generation unit 205 at a predetermined time interval. The camera selection information generation unit 205 generates camera selection information based on the position of the input device 20 input from the position identification unit 204. The camera selection information generation unit 205 transmits the generated camera selection information to the center system 10.

[0077] Each time the camera selection unit 105 determines in the process of step S4 that it has received camera selection information, it may perform the processes of step S5 and onward.

[0078] In addition, when the storage unit 106 of the image processing device 100 stores predetermined conditions such as cameras installed within 1 kilometer from the position of the input device 20 or the camera installed closest to the position of the input device 20, the input device 20 may not include the camera selection information generation unit 205 and may transmit the position of the input device 20 identified by the position identification unit 204. Also, the center system 10 may directly acquire the position of the input device 20 from a GPS. In these cases, the camera selection unit 105 of the image processing device 100 may identify an edge device based on the acquired position of the input device 20 and the predetermined conditions stored in the storage unit 106.

[0079] The image processing system 1 according to the second embodiment of the present invention has been described above. In the image processing device 100 of the above-mentioned image processing system 1, the appearance state determination unit 101 determines the appearance state of a person in a captured video based on a person candidate determination condition. The predetermined person candidate estimation unit 102 estimates a predetermined person candidate based on the appearance state determined by the appearance state determination unit 101. This makes it possible to identify suspected individuals based on specified criteria, such as suspicious people.

[0080] Furthermore, the position identification unit 204 identifies the position of the input device 20 at predetermined time intervals while the user carrying the input device 20 is moving. The position identification unit 204 outputs the identified position of the input device 20 to the camera selection information generation unit 205 at predetermined time intervals. The camera selection information generation unit 205 generates camera selection information based on the position of the input device 20 input from the position identification unit 204. The camera selection information generation unit 205 transmits the generated camera selection information to the center system 10. Every time the camera selection unit 105 determines that it has received the camera selection information generated by the camera selection information generation unit 205, it performs the processes from step S5 onwards. This makes it possible to cope with a case where it is necessary to change the camera capturing the appearance state of a person in the captured video from time to time, such as when monitoring a stalker.

[0081] In the embodiment of the present invention, as long as information is appropriately transmitted and received, a part or all of the functional units of the image processing device 100 may be provided in any one of the first edge device 30a, the second edge device 30b, the third edge device 30c, ..., the n-th edge device 30n. Also, a part or all of the functional units of the image processing device 100 may be distributed and provided in any one of the first edge device 30a, the second edge device 30b, the third edge device 30c, ..., the n-th edge device 30n.

[0082] In addition, in an embodiment of the present invention, as long as information is transmitted and received appropriately, some or all of the functional units of the first edge device 30a, the second edge device 30b, the third edge device 30c, ..., the nth edge device 30n may be provided in the center system 10.

[0083] In the embodiment of the present invention, when the input device 20 may be installed at any fixed location (for example, when the position of the input device 20 is unrelated to the position of the camera that takes the image as in the first embodiment), as long as the transmission and reception of information is performed appropriately, a part or all of the functional units of the input device 20 may be provided in the center system 10. In addition, in the embodiment of the present invention, when the input device 20 may be installed at any fixed location, as long as the transmission and reception of information is performed appropriately, a part or all of the functional units of the input device 20 may be provided in any of the first edge device 30a, the second edge device 30b, the third edge device 30c, ..., the n-th edge device 30n. In the embodiment of the present invention, when the input device 20 may be installed at any fixed location, as long as the transmission and reception of information is performed appropriately, a part or all of the functional units of the input device 20 may be distributed and provided in any of the center system 10, the first edge device 30a, the second edge device 30b, the third edge device 30c, ..., the n-th edge device 30n.

[0084] In addition, in an embodiment of the present invention, as long as information is transmitted and received appropriately, the image processing device 100 may include one or more of the first feature extraction unit 305a, the second feature extraction unit 305b, the third feature extraction unit 305c,..., the n-th feature extraction unit 305n.

[0085] In addition, each of the storage unit 106, the first storage unit 304a, the second storage unit 304b, and the third storage unit 304c in the present invention may be provided anywhere within a range where appropriate information is transmitted and received. Also, each of the storage unit 106, the first storage unit 304a, the second storage unit 304b, and the third storage unit 304c may be present in a plurality of units within a range where appropriate information is transmitted and received, and data may be stored in a distributed manner.

[0086] In the process flow according to the embodiment of the present invention, the order of processes may be changed as long as appropriate processes are performed.

[0087] Although the embodiment of the present invention has been described, each of the image processing device 100, the input device 20, the first edge device 30a, the second edge device 30b, and the third edge device 30c has a computer system therein. The above-mentioned process steps are stored in a computer-readable storage unit in the form of a program, and the computer reads and executes the program to perform the above-mentioned process. Here, the computer-readable storage unit refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Also, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program.

[0088] The program may be for implementing some of the functions described above, or may be a so-called differential file (differential program) that can implement the functions described above in combination with a program already recorded in the computer system.

[0089] Although several embodiments of the present invention have been described, these embodiments are presented as examples and do not limit the scope of the invention. Furthermore, various omissions, substitutions, and modifications can be made without departing from the gist of the invention. [Explanation of symbols]

[0090] 1. Image Processing System 10. Center System 20. Input device 30a First edge device 30b Second edge device 30c Third edge device 40a...First camera device 40b Second camera device 40c Third camera device 100...Image processing device 101: Appearance state determination unit 102...Predetermined person candidate estimation unit 103, 201...Communications Department 104...Control section 105 Camera selection section 106...Storage section 107 Condition input section 202 Information input section 203 Display control unit 204...Location specifying section 205 Camera selection information generation unit 301a...First Communication Department 301b...Second Communication Department 301c...Third Communication Department 302a First control section 302b Second control section 302c Third control section 303a First camera control unit 303b Second camera control unit 303c Third camera control unit 304a...First storage section 304b...Second storage section 304c...Third memory section 305a First feature extraction unit 305b Second feature extraction unit 305c Third feature extraction unit 306a First video receiving unit 306b Second video receiving unit 306c Third video receiving unit 307a First camera interface 307b Secondary camera interface 307c···Third camera interface

Claims

1. A system including a terminal and an information processing device capable of communicating with the terminal, The terminal includes: a determination unit that determines whether a person appearing in an image captured by a camera disposed at a cash register is a predetermined person based on whether the face of the person is extracted from the image; a transmission unit that transmits the image showing the person and information about the person to the information processing device, The information processing device includes: a receiving unit that receives the person's information and the video; a storage unit that stores the person information and the video; a transmitting unit that transmits predetermined information to a terminal different from the terminal when the predetermined information is received; An information processing system comprising:

2. The information about the person is a person feature amount that is information for identifying the person. The information processing system according to claim 1 .

3. The predetermined information is data that indicates the appearance of the person determined to be the predetermined person.

3. The information processing system according to claim 1 or 2.

4. The information processing system according to claim 1 , wherein the predetermined information is data captured by a camera.

5. The camera includes: The camera is selected from among multiple cameras installed in multiple stores. The information processing system according to claim 1 .

6. An information processing device capable of communicating with a terminal including a determination unit that determines whether a person appearing in an image captured by a camera disposed at a cash register is a predetermined person based on whether the face of the person is extracted from the image, the information processing device comprising: A receiving unit that receives an image captured by a camera in which a person who has performed a predetermined behavior is shown and information about the person; a storage unit that stores the person information and the video; a transmitting unit that transmits predetermined information to a terminal different from the terminal when the predetermined information is received; Equipped Information processing device.

7. The information about the person is a person feature amount that is information for identifying the person. The information processing device according to claim 6.

8. The predetermined information is data that indicates the appearance of the person determined to be the predetermined person.

8. The information processing device according to claim 6 or 7.

9. The information processing apparatus according to claim 6 , wherein the predetermined information is data captured by a camera.

10. The camera includes: The camera is selected from among multiple cameras installed in multiple stores. The information processing device according to claim 6 .

11. A terminal capable of communicating with an information processing device, The terminal includes: a determination unit that determines whether a person appearing in an image captured by a camera disposed at a cash register is a predetermined person based on whether the face of the person is extracted from the image; a transmission unit that transmits the image showing the person and information about the person to the information processing device, The information processing device includes: a receiving unit that receives the person's information and the video; a storage unit that stores the person information and the video; A transmitting unit that transmits predetermined information to a terminal different from the terminal when the reception is performed. Terminal.

12. The camera includes: The camera is selected from among multiple cameras installed in multiple stores. The terminal according to claim 11.

13. A processing method executed by a system including a terminal and an information processing device capable of communicating with the terminal, The terminal includes: determining whether a face of a person appearing in an image captured by a camera disposed at the cash register is extracted from the image, and Transmitting the image showing the person and information about the person to the information processing device; The information processing device includes: receiving information about the person and the image; storing information about the person and the image; When the reception is performed, the predetermined information is transmitted to a terminal different from the terminal. Processing methods.

14. A processing method executed by an information processing device capable of communicating with a terminal including a determination unit that determines that a person appearing in an image captured by a camera disposed at a cash register is a predetermined person based on whether or not the face of the person is extracted from the image, the method comprising: Receiving a video of a person who has performed a predetermined behavior in a video captured by a camera and information about the person; storing information about the person and the image; When the reception is performed, transmitting predetermined information to a terminal different from the terminal. Processing methods.

15. A processing method executed by a terminal capable of communicating with an information processing device, determining whether a face of a person appearing in an image captured by a camera disposed at a cash register is extracted from the image, and transmitting the image showing the person and information about the person to the information processing device; The information processing device includes: a receiving unit that receives the person's information and the video; a storage unit that stores the person information and the video; A transmitting unit that transmits predetermined information to a terminal different from the terminal when the reception is performed. Processing methods.

16. The camera includes: The camera is selected from among multiple cameras installed in multiple stores.

16. The method according to any one of claims 13 to 15.

17. A computer of an information processing device capable of communicating with a terminal including a determination unit that determines whether a person appearing in an image captured by a camera disposed at a cash register is a predetermined person based on whether the face of the person is extracted from the image, Receiving a video of a person who has performed a predetermined behavior in a video captured by a camera and information about the person; storing information about the person and the image; When the reception is performed, the predetermined information is transmitted to a terminal different from the terminal. program.

18. A terminal computer capable of communicating with the information processing device determining whether a face of a person appearing in an image captured by a camera disposed at a cash register is extracted from the image, and transmitting the image showing the person and information about the person to the information processing device; The information processing device includes: a receiving unit that receives the person's information and the video; a storage unit that stores the person information and the video; A transmitting unit that transmits predetermined information to a terminal different from the terminal when the reception is performed. program.

19. The camera includes: The camera is selected from among multiple cameras installed in multiple stores.

19. The program according to claim 17 or 18.

Images