Information processing system and processing method

Abstract

To provide an image processing device capable of monitoring a suspicious person even when the suspicious person is not known.SOLUTION: An image processing system comprises: a detection unit which with respect to a person appearing in a video captured from a camera selected based on an input to an input device, detects a gesture of the person; and an estimation unit which estimates a predetermined person candidate based on a result of the detection by the detection unit.SELECTED DRAWING: Figure 1

Description

[[Technical Field]]

[0001] The present invention relates to an information processing system, and processing method . [[Background Art]]

[0002] The installation of security cameras is increasing in condominiums, single-family homes, urban areas, and the like. Patent Document 1 describes a technique for monitoring suspicious persons as a related technique. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2010-191620 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] Incidentally, the technique described in Patent Document 1 is based on the premise that a suspicious person is known in advance, and cannot monitor a suspicious person when the suspicious person is not known.

[0005] Accordingly, an object of the present invention is to provide an information processing system capable of solving the above problem, and processing method . [[Means for Solving the Problem]]

[0006] According to one aspect of the present disclosure, an information processing system includes: The edge device comprises: a first transmitting unit of an edge device that transmits person information, which is information about a person appearing in a video captured by a camera selected from among multiple cameras installed in multiple stores, and the video to a central system; a receiving unit of the central system that receives the person information and the video from the first transmitting unit; a storage unit of the central system that stores the person information and the video received by the receiving unit, and a determination condition; a determination unit of the central system that determines the person to be a predetermined person based on the determination condition stored in the storage unit when the person performs a predetermined action; and a second transmitting unit of the central system that transmits predetermined information to an external device when the receiving unit receives the person information and the video and the determination unit determines the person to be the predetermined person. .

[0008] Further, according to one aspect of the present disclosure, a processing method includes: The edge device transmits to a central system person information, which is information about a person appearing in a video captured by a camera selected from among multiple cameras installed in multiple stores, and the video itself; the central system receives the person information and the video from the edge device; the central system stores the received person information and video along with a determination condition; the central system determines that the person is a predetermined person based on the determination condition when the person performs a predetermined action; and the central system transmits predetermined information to an external device when it receives the person information and the video and determines that the person is the predetermined person. . [[Advantageous Effects of the Invention]]

[0010] The image processing system of the present invention can identify potential individuals based on specified conditions, such as suspicious persons. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows the minimum configuration of an image processing device according to an embodiment of the present invention. [Figure 2] This figure shows the configuration of an image processing system according to the first embodiment of the present invention. [Figure 3] This diagram shows the configuration of the center system according to this embodiment. [Figure 4] This figure shows the configuration of the input device according to this embodiment. [Figure 5] This figure shows the configuration of the first edge device according to this embodiment. [Figure 6] This figure shows the configuration of the second edge device according to this embodiment. [Figure 7] This figure shows the configuration of the third edge device according to this embodiment. [Figure 8] This figure shows the first data table according to this embodiment. [Figure 9] This figure shows the second data table according to this embodiment. [Figure 10] This figure shows the third data table according to this embodiment. [Figure 11] This figure shows the comprehensive data table according to this embodiment. [Figure 12] This is the first diagram showing the processing flow according to this embodiment. [Figure 13] This is the second diagram showing the processing flow according to this embodiment. [Figure 14] This figure shows the configuration of an input device according to the first embodiment of the present invention. [Modes for carrying out the invention]

[0012] Figure 1 shows the minimum configuration of the image processing apparatus 100 according to an embodiment of the present invention. The image processing apparatus 100 of the present invention comprises at least an appearance state determination unit 101 and a predetermined person candidate estimation unit 102, as shown in FIG. 1. The appearance state determination unit 101 determines the appearance state of a person in a captured video. The appearance state of a person refers to a result indicating the appearance of a person, for example, cases where the person exhibits suspicious behavior such as appearing many times in the video, appearing for a long time, or frequently looking around the surroundings. The predetermined person candidate estimation unit 102 estimates a predetermined person candidate based on the appearance state. The predetermined person candidate is, for example, a person such as a shoplifter, a suspicious purchaser, or a stalker.

[0013] <First Embodiment> FIG. 2 is a diagram showing the configuration of an image processing system 1 according to the first embodiment of the present invention. First, an image processing system 1 including the image processing apparatus 100 according to the first embodiment of the present invention will be described. As shown in FIG. 2, the image processing system 1 according to the present embodiment comprises a center system 10, an input device 20, a first edge device 30a (information processing apparatus), a second edge device 30b (information processing apparatus), a third edge device 30c (information processing apparatus), a first camera device 40a, a second camera device 40b, and a third camera device 40c. In the following description, the first edge device 30a, the second edge device 30b, the third edge device 30c, ..., the n-th edge device 30n, ... are collectively referred to as the edge device 30. Further, the first camera device 40a, the second camera device 40b, the third camera device 40c, ..., the n-th camera device 40n, ... are collectively referred to as the camera device 40.

[0014] The input device 20 is, for example, a smartphone, a mobile phone, a personal computer, or the like. A user performs a camera selection information input operation for inputting camera selection information for selecting a camera from which a person feature amount, which is a feature amount of a person, is to be extracted. The person feature amount is information for specifying a person, such as a face, height, gender, clothing, belongings, and the like, for example.

[0015] The user performs a start command input operation to enter a start command to begin extracting human features. Based on the user's start command input operation, the input device 20 transmits camera selection information to the central system 10.

[0016] The central system 10 receives camera selection information from the input device 20. Based on the input camera selection information, the central system 10 identifies the edge device. The central system 10 transmits camera selection information and a feature extraction start instruction to each of the identified first edge device 30a, second edge device 30b, and third edge device 30c, instructing them to start extracting human features.

[0017] When the first edge device 30a, the second edge device 30b, and the third edge device 30c each receive camera selection information and a human feature extraction start instruction from the center system 10, they acquire camera images from the camera device 40 that has the camera indicated by the camera selection information. Then, the first edge device 30a, the second edge device 30b, and the third edge device 30c each extract human features from the acquired camera images. The first edge device 30a, the second edge device 30b, and the third edge device 30c each generate an index for the extracted human features. The index includes information such as the input device ID (IDentifier), which is the identifier of the input device 20 that transmitted the camera selection information, and the human feature ID, which is the identifier of the human feature assigned to each human feature. The first edge device 30a, the second edge device 30b, and the third edge device 30c each transmit the extracted human features along with the generated index to the center system 10. Furthermore, the first edge device 30a, the second edge device 30b, and the third edge device 30c each record the generated index along with the extracted person features in their respective memory units. Additionally, the first edge device 30a, the second edge device 30b, and the third edge device 30c each assign the generated index to the camera image and record the camera image in their memory units.

[0018] When the center system 10 receives camera images along with person features and indices from the first edge device 30a, the second edge device 30b, and the third edge device 30c, it associates the received camera images with the person features and indices and records them in the storage unit.

[0019] Furthermore, the user performs a suspicious person extraction start instruction input operation, which involves entering a suspicious person extraction start instruction to initiate the extraction of suspicious individuals. Based on the user's suspicious person extraction start instruction, the input device 20 transmits the start of suspicious person extraction to the central system 10.

[0020] The central system 10 receives and records in its memory the conditions for determining whether a person is a suspect or not. These conditions determine the appearance status, which indicates the presence of a person, such as a high number of appearances or a long appearance in the video. When the central system 10 receives a suspicious person extraction start instruction to begin extracting suspicious persons, it identifies a suspicious person based on the camera footage, person features and index stored in the memory, as well as the appearance status. The central system 10 then records the results of the identified suspicious person in its memory.

[0021] The user performs a termination command input operation to enter a termination command to end the extraction of person features. Based on the user's termination command input operation, the input device 20 transmits the termination command to the central system 10.

[0022] When the center system 10 receives a termination instruction from the input device 20, it transmits a feature extraction termination instruction to the first edge device 30a, the second edge device 30b, and the third edge device 30c, respectively, indicating the completion of the extraction of human features.

[0023] When the first edge device 30a, the second edge device 30b, and the third edge device 30c each receive a feature extraction termination instruction from the center system 10, they terminate the extraction of human features from the camera image.

[0024] As described above, in the image processing system 1 according to the embodiment of the present invention, the camera device 40 captures video in the area selected by the user. Each of the edge devices 30 extracts human features from the video captured by the camera device 40. The center system 10 identifies individuals who satisfy the appearance conditions determined using the human candidate determination conditions as suspicious individuals from among the individuals with human features extracted by the camera device 40.

[0025] Figure 3 shows the configuration of the center system 10 according to this embodiment. Next, the configuration of the center system 10 according to this embodiment will be described. As shown in Figure 3, the center system 10 according to this embodiment includes an image processing device 100. The image processing device 100 includes an appearance state determination unit 101, a predetermined person candidate estimation unit 102, a communication unit 103, a control unit 104, a camera selection unit 105, a storage unit 106, and a condition input unit 107.

[0026] The appearance status determination unit 101 determines the appearance status of a person in the captured video based on the person candidate determination conditions. The predetermined person candidate estimation unit 102 estimates predetermined person candidates based on the appearance status of the person in the captured video. For example, the predetermined person candidate estimation unit 102 identifies a person in the camera video that has person features satisfying the appearance status as a suspicious person, based on the camera video, person features, index, and appearance status received by the center system 10 from each of the edge devices 30. The communication unit 103 transmits and receives information with the input device 20 and the edge device 30, respectively.

[0027] The control unit 104 performs various controls necessary for the processing carried out by the center system 10. For example, the control unit 104 controls the transmission and reception of information performed by the communication unit 103. In the following description, even if it is not specifically stated that the control unit 104 controls the transmission and reception of information performed by the communication unit 103, the control unit 104 does control the transmission and reception of information performed by the communication unit 103.

[0028] The camera selection unit 105 specifies the camera to capture the appearance status of a person in the captured video, as determined by the appearance status determination unit 101. The memory unit 106 stores information necessary for various processes performed by the central system 10. For example, the memory unit 106 stores camera images received by the central system 10 from each of the edge devices 30, associating them with person features and indices. The condition input unit 107 acquires the conditions for determining a candidate person and records the acquired conditions in the storage unit 106.

[0029] Figure 4 shows the configuration of the input device 20 according to this embodiment. Next, the configuration of the input device 20 according to this embodiment will be described. As shown in Figure 4, the input device 20 according to this embodiment comprises a communication unit 201, an information input unit 202, and a display control unit 203.

[0030] The communications unit 201 transmits and receives information with the central system 10. The information input unit 202 detects user input operations and outputs signals based on the detected input operations. For example, if the information input unit 202 detects a user input of camera selection information, it transmits the camera selection information to the central system 10. Also, for example, if the information input unit 202 detects a user input of a start command, it transmits a start command to the central system 10. Also, for example, if the information input unit 202 detects a user input of an end command, it transmits an end command to the central system 10.

[0031] The display control unit 203 controls the display shown on a display screen (not shown). For example, the display control unit 203 controls the display shown on the display screen for information transmitted by the input device 20 to the central system 10 and information received by the input device 20 from the central system 10.

[0032] Next, the configuration of the edge device 30 according to this embodiment will be described. Figure 5 shows the configuration of the first edge device 30a according to this embodiment. First, the configuration of the first edge device 30a according to this embodiment will be described. As shown in Figure 5, the first edge device 30a according to this embodiment includes a first communication unit 301a, a first control unit 302a, a first camera control unit 303a, a first storage unit 304a, a first feature extraction unit 305a, a first video receiving unit 306a, and a first camera interface 307a.

[0033] The first communication unit 301a transmits and receives information with the center system 10 and the first camera device 40a, respectively. The first control unit 302a performs various controls necessary for the processing carried out by the first edge device 30a. For example, the first control unit 302a controls the transmission and reception of information performed by the first communication unit 301a. In the following description, even if it is not specifically stated that the first control unit 302a controls the transmission and reception of information performed by the first communication unit 301a, the first control unit 302a does control the transmission and reception of information performed by the first communication unit 301a.

[0034] The first camera control unit 303a transmits a shooting control signal to the first camera device 40a via the first camera interface 307a to control shooting. The first storage unit 304a stores various information necessary for the processing performed by the first edge device 30a.

[0035] The first feature extraction unit 305a extracts human features captured by the camera of the camera device 40 specified by the camera selection unit 105. Specifically, the first video receiving unit 306a performs image analysis on the video received from the first camera device 40a to extract human features.

[0036] The first video receiving unit 306a receives the video captured by the first camera device 40a. The first camera interface 307a is an interface provided by the first edge device 30a to connect the first edge device 30a and the first camera device 40a.

[0037] Figure 6 shows the configuration of the second edge device 30b according to this embodiment. Next, the configuration of the second edge device 30b according to this embodiment will be described. As shown in Figure 6, the second edge device 30b according to this embodiment includes a second communication unit 301b, a second control unit 302b, a second camera control unit 303b, a second storage unit 304b, a second feature extraction unit 305b, a second video receiving unit 306b, and a second camera interface 307b.

[0038] The second communication unit 301b transmits and receives information to and from the center system 10 and the second camera device 40b, respectively. The second control unit 302b performs various controls necessary for the processing carried out by the second edge device 30b. For example, the second control unit 302b controls the transmission and reception of information performed by the second communication unit 301b. In the following description, even if it is not specifically stated that the second control unit 302b controls the transmission and reception of information performed by the second communication unit 301b, the second control unit 302b does control the transmission and reception of information performed by the second communication unit 301b.

[0039] The second camera control unit 303b transmits a shooting control signal to the second camera device 40b via the second camera interface 307b to control the shooting process. The second memory unit 304b stores various information necessary for the processing performed by the second edge device 30b.

[0040] The second feature extraction unit 305b extracts human features captured by the camera of the camera device 40 specified by the camera selection unit 105. Specifically, the second video receiving unit 306b performs image analysis on the video received from the second camera device 40b to extract human features.

[0041] The second video receiving unit 306b receives the video captured by the second camera device 40b. The second camera interface 307b is an interface provided by the second edge device 30b to connect the second edge device 30b and the second camera device 40b.

[0042] Figure 7 shows the configuration of the third edge device 30c according to this embodiment. Next, the configuration of the third edge device 30c according to this embodiment will be described. As shown in Figure 7, the third edge device 30c according to this embodiment includes a third communication unit 301c, a third control unit 302c, a third camera control unit 303c, a third storage unit 304c, a third feature extraction unit 305c, a third video receiving unit 306c, and a third camera interface 307c.

[0043] The third communication unit 301c transmits and receives information with the center system 10 and the third camera device 40c, respectively. The third control unit 302c performs various controls necessary for the processing carried out by the third edge device 30c. For example, the third control unit 302c controls the transmission and reception of information performed by the third communication unit 301c. In the following description, even if it is not specifically stated that the third control unit 302c controls the transmission and reception of information performed by the third communication unit 301c, the third control unit 302c does control the transmission and reception of information performed by the third communication unit 301c.

[0044] The third camera control unit 303c transmits a shooting control signal to the third camera device 40c via the third camera interface 307c to control the shooting process. The third memory unit 304c stores various information necessary for the processing performed by the third edge device 30c.

[0045] The third feature extraction unit 305c extracts human features captured by the camera of the camera device 40 specified by the camera selection unit 105. For example, the third video receiving unit 306c performs image analysis on the video received from the third camera device 40c to extract features such as face, height, gender, clothing, and belongings.

[0046] The third video receiving unit 306c receives the video captured by the third camera device 40c. The third camera interface 307c is an interface provided by the third edge device 30c to connect the third edge device 30c and the third camera device 40c.

[0047] Next, the configuration of the camera device 40 according to this embodiment will be described. Each of the camera devices 40 is equipped with one or more cameras. In this embodiment, the first camera device 40a is equipped with cameras 401a, 402a, and 403a. The second camera device 40b is equipped with cameras 401b, 402b, and 403b. The third camera device 40c is equipped with cameras 401c and 402c.

[0048] Next, we will describe the data tables stored in the first edge device 30a, the second edge device 30b, and the third edge device 30c, respectively. Figure 8 shows the first data table TBL1 according to this embodiment. The first storage unit 304a stores the person features extracted by the first feature extraction unit 305a in the first data table TBL1. For example, as shown in Figure 8, the first storage unit 304a stores the person features extracted by the first feature extraction unit 305a from the images captured by cameras 401a, 402a, and 403a, associating each camera with a date and time and an index. The capture date and time in the first data table TBL1 shown in Figure 8 is the date and time when cameras 401a, 402a, and 403a each began capturing the portion of the image from which the first feature extraction unit 305a extracted person features. The release date and time in the first data table TBL1 shown in Figure 8 is the date and time when cameras 401a, 402a, and 403a each finished capturing the portion of the image from which the first feature extraction unit 305a extracted person features. Furthermore, the index includes information such as the input device ID, which is the identifier of the input device 20 that transmitted the camera selection information, and the person feature ID, which is the identifier of the person feature assigned to each person feature.

[0049] Figure 9 shows the second data table TBL2 according to this embodiment. The second memory unit 304b stores the person features extracted by the second feature extraction unit 305b in the second data table TBL2. For example, as shown in Figure 9, the second memory unit 304b stores the person features extracted by the second feature extraction unit 305b from the images captured by cameras 401b, 402b, and 403b, associating each camera with a date and time and an index.

[0050] Figure 10 shows the third data table TBL3 according to this embodiment. The third memory unit 304c stores the person features extracted by the third feature extraction unit 305c in the third data table TBL3. For example, as shown in Figure 10, the third memory unit 304c stores the person features extracted by the third feature extraction unit 305c from the images captured by camera 401c and camera 402c, associating each camera with a date and time and an index.

[0051] Next, we will explain the comprehensive data table STBL stored in the central system 10. Figure 11 shows the integrated data table STBL according to this embodiment. The memory unit 106 (person feature storage unit) stores the person features extracted by the first feature extraction unit 305a, the second feature extraction unit 305b, and the third feature extraction unit 305c in the integrated data table STBL. For example, the memory unit 106 stores the integrated data table STBL, as shown in Figure 11, which combines the first data table TBL1, the second data table TBL2, and the third data table TBL3 into a single data table.

[0052] Figures 12 and 13 show the processing flow of the image processing system 1 according to this embodiment. Next, the processing of the image processing system 1 according to this embodiment will be described. This section describes the process by which image processing system 1 monitors suspicious individuals at multiple stores of the same chain. The first edge device 30a and the first camera device 40a are assumed to be located at the first store A. The second edge device 30b and the second camera device 40b are assumed to be located at the second store B. The third edge device 30c and the third camera device 40c are assumed to be located at the third store C. Furthermore, the following explanation uses the example where an employee of the first store A is the user and the person's feature is a face. Although the first edge device 30a, the second edge device 30b, and the third edge device 30c each perform information processing independently, their operation is similar, so in the processing flows shown in Figures 12 and 13, they are referred to as edge device 30.

[0053] First, we will explain the process of extracting human features performed by the image processing system 1 according to this embodiment, as shown in the processing flow of Figure 12. The user performs a camera selection information input operation on the input device 20 to select a camera installed in multiple stores of the same chain from which to extract faces. The information input unit 202 inputs camera selection information in response to the user's camera selection information input operation (step S1). The camera selection information indicates that all cameras installed in the first store A, the second store B, and the third store C are selected. The cameras installed in the first store A are cameras 401a, 402a, and 403a. The cameras installed in the second store B are cameras 401b, 402b, and 403b. The cameras installed in the third store C are cameras 401c and 402c. The information input unit 202 then determines whether or not to start face extraction (step S2). If the information input unit 202 does not detect a start instruction input operation to input a start instruction to start face extraction, it returns to the process in step S2.

[0054] The user performs a start command input operation on the input device 20 to input a start command to begin face extraction. The information input unit 202 detects a start command input operation by the user (step S2, YES) and transmits camera selection information to the central system 10 (step S3).

[0055] The camera selection unit 105 determines whether or not to receive camera selection information (step S4). If the camera selection unit 105 does not receive camera selection information, it returns to the process in step S4. When the camera selection unit 105 receives camera selection information from the input device 20 (step S4, YES), it identifies the edge device based on the received camera selection information (step S5). Specifically, the camera selection unit 105 identifies the first edge device 30a corresponding to cameras 401a, 402a, and 403a included in the camera selection information, the second edge device 30b corresponding to cameras 401b, 402b, and 403b, and the third edge device 30c corresponding to cameras 401c and 402c. For example, the camera selection unit 105 identifies the matching camera by sequentially comparing the camera in the data table showing the correspondence between cameras and edge devices with the camera included in the received camera selection information. The camera selection unit 105 identifies the edge device corresponding to the matching camera as the selected edge device.

[0056] The camera selection unit 105 transmits camera selection information and a feature extraction start instruction to each of the identified first edge device 30a, second edge device 30b, and third edge device 30c, respectively, instructing them to start extracting human features (step S6).

[0057] The first video receiving unit 306a determines whether or not it has received an instruction to start extracting human features (step S7), and determines whether or not to start extracting human features based on the result of that determination.

[0058] If the first video receiving unit 306a determines that it has not received an instruction to start extracting human features (step S7, NO), it determines not to start extracting human features. In this case, the first video receiving unit 306a receives camera images from cameras 401a, 402a, and 403a respectively (step S8), and records the received camera images in the first storage unit 304a (step S9).

[0059] Furthermore, if the first video receiving unit 306a determines that it has received an instruction to start extracting human features (step S7, YES), it determines to start extracting human features. In this case, the first video receiving unit 306a receives camera images from cameras 401a, 402a, and 403a, respectively, as indicated by the camera selection information (step S10). The first feature extraction unit 305a extracts faces from each camera image received by the first video receiving unit 306a (step S11). The first feature extraction unit 305a generates an index for the extracted faces (step S12). The first feature extraction unit 305a transmits the index generated along with the extracted faces to the central system 10 (step S13). Furthermore, the first feature extraction unit 305a records the index generated along with the face information of the extracted faces in the first data table TBL1 of the first storage unit 304a (step S14). The first feature extraction unit 305a also assigns the generated index to each camera image (step S15) and records each camera image in the first storage unit 304a (step S16).

[0060] Similar to the first video receiver 306a, the second video receiver 306b determines to start extracting human features if it has received an instruction to start extracting human features (step S7, YES). In this case, the second video receiver 306b receives camera images from cameras 401b, 402b, and 403b, respectively, as indicated by the camera selection information (step S10). The second feature extraction unit 305b extracts faces from each camera image received by the second video receiving unit 306b (step S11). The second feature extraction unit 305b generates an index for the extracted faces (step S12). The second feature extraction unit 305b transmits the extracted faces along with the generated index to the central system 10 (step S13). Furthermore, the second feature extraction unit 305b records the index generated along with the face information of the extracted faces in the second data table TBL2 of the second storage unit 304b (step S14). The second feature extraction unit 305b also assigns the generated index to each camera image (step S15) and records each camera image in the second storage unit 304b (step S16).

[0061] The third video receiver 306c, like the first video receiver 306a, determines to start extracting human features if it has received an instruction to start extracting human features (step S7, YES). In this case, the third video receiver 306c receives camera images from camera 401c and camera 402c, respectively, as indicated by the camera selection information (step S10). The third feature extraction unit 305c extracts faces from each camera image received by the third video receiving unit 306c (step S11). The third feature extraction unit 305c generates an index for the extracted faces (step S12). The third feature extraction unit 305c transmits the extracted faces along with the generated index to the central system 10 (step S13). Furthermore, the third feature extraction unit 305c records the index generated along with the face information of the extracted faces in the third data table TBL3 of the third storage unit 304c (step S14). The third feature extraction unit 305c also assigns the generated index to each camera image (step S15) and records each camera image in the third storage unit 304c (step S16).

[0062] When the appearance state determination unit 101 receives face information and an index from the first edge device 30a, the second edge device 30b, and the third edge device 30c, respectively (step S17), it associates the received face information and index and records them in the comprehensive data table STBL of the storage unit 106 (step S18). In this case, the center system 10 can reduce the amount of communication data by receiving only face information (personal features) and an index, without receiving camera images from the first edge device 30a, the second edge device 30b, and the third edge device 30c. However, the center system 10 may also receive camera images along with face information (personal features) and an index from the first edge device 30a, the second edge device 30b, and the third edge device 30c. Alternatively, the center system 10 may record the camera images received from the first edge device 30a, the second edge device 30b, and the third edge device 30c in a storage unit.

[0063] Afterward, the user performs a termination command input operation, which involves entering a termination command to end the extraction of human features. The information input unit 202 inputs a termination instruction based on the user's termination instruction input operation (step S19). The information input unit 202 transmits the termination instruction to the central system 10 (step S20).

[0064] The appearance status determination unit 101 constantly determines whether or not to terminate the extraction of human features (step S21). If the appearance status determination unit 101 does not receive a termination instruction from the input device 20 (step S21, NO), it returns to the process in step S17. Furthermore, if the appearance state determination unit 101 receives a termination instruction from the input device 20 (step S21, YES), it transmits a feature extraction termination instruction to the first edge device 30a, the second edge device 30b, and the third edge device 30c, respectively, indicating the completion of the extraction of human features (step S22).

[0065] The first feature extraction unit 305a determines whether or not it has received a feature extraction termination instruction from the center system 10 (step S23), and determines whether or not to terminate face extraction in the camera image based on the determination result.

[0066] If the first feature extraction unit 305a determines that it has not received a feature extraction termination instruction (step S23, NO), it determines that it will not terminate the extraction of faces from the camera image. In this case, the first feature extraction unit 305a returns to the process of step S10.

[0067] Furthermore, if the first video receiver 306a determines that it has received a feature extraction termination instruction (step S23, YES), it determines to terminate the extraction of faces from the camera image. In this case, the first video receiver 306a returns to the process in step S8. The above describes the human feature extraction process performed by the image processing system 1 according to this embodiment.

[0068] Next, we will explain the process performed by the image processing system 1 according to this embodiment, as shown in the processing flow of Figure 13, to estimate a predetermined person candidate. After the process of extracting human features, the user who operated the input device 20, or another user who identifies a suspicious person (for example, a police officer), performs a suspicious person extraction start instruction input operation to the center system 10, either directly or via a device installed remotely and connected by communication, to input a suspicious person extraction start instruction that signals the start of the extraction of suspicious persons.

[0069] The memory unit 106 has pre-stored criteria for determining a person as a suspect, which have been entered via the condition input unit 107. For example, the criteria for determining a person as a suspect include identifying a person who is likely to shoplift, such as repeatedly entering and leaving the same store on the same day, staying in the same store for a long time on the same day but not purchasing any goods, repeatedly entering and leaving multiple stores of the same chain on the same day, staying in multiple stores of the same chain for a long time on the same day but not purchasing any goods, or frequently visiting multiple stores of the same chain over several days but not purchasing any goods. In addition, the criteria for determining a person as a suspect include identifying a person who is likely to make large purchases, such as the number of times a person appears and the time of appearance, which are determined based on factors such as repeatedly entering and leaving the same store on the same day and purchasing goods multiple times, repeatedly entering and leaving multiple stores of the same chain on the same day and purchasing goods multiple times, and frequently visiting each store over several days and purchasing goods multiple times.

[0070] When the predetermined person candidate estimation unit 102 receives an instruction to start extracting suspicious persons (step S31), it identifies a suspicious person based on the camera images, face information, index, and appearance status stored in the memory unit 106 (step S32). Specifically, the predetermined person candidate estimation unit 102 sorts the data for each face information (feature quantity) in the comprehensive data table STBL stored in the memory unit 106, identifies the number of occurrences of each face information, the insertion time of each face information, etc., and identifies the person corresponding to the face information as a suspicious person if the person candidate determination conditions are met. The predetermined person candidate estimation unit 102 determines that the person whose face is extracted has purchased a product if a face is extracted in the camera image of the camera placed at the register. The predetermined person candidate estimation unit 102 determines that the person whose face is extracted has not purchased a product if a face is not extracted in the camera image of the camera placed at the register. Furthermore, the designated person candidate estimation unit 102 counts the number of times a person's face appears in the camera footage from a camera installed at the entrance of the store, based on the number of times that face has been extracted. The designated person candidate estimation unit 102 records the camera footage and index, which are the results of identifying the identified suspicious person, in the storage unit 106 (step S33). The above describes the process performed by the image processing system 1 according to this embodiment to estimate a predetermined person candidate. If the designated person candidate estimation unit 102 identifies a suspicious person, it may transmit camera video data that reveals the appearance of the suspicious person, such as their face, height, gender, clothing, and belongings, to the input device 20 based on the camera video from which the suspicious person's characteristics have been extracted. In this case, the display control unit 203 displays the camera video data revealing the appearance of the suspicious person from the center system 10 on the display unit, allowing the user to confirm the suspicious person. The designated person candidate estimation unit 102 may also provide the camera video data revealing the appearance of the suspicious person to the Metropolitan Police Department or other relevant authorities. In this case, it can be used to help prevent crime.

[0071] The image processing system 1 according to the first embodiment of the present invention has been described above. In the image processing apparatus 100 of the image processing system 1 described above, the appearance state determination unit 101 determines the appearance state of a person in the captured video. The predetermined person candidate estimation unit 102 estimates predetermined person candidates based on the appearance state determined by the appearance state determination unit 101. This allows for the identification of individuals who meet specified criteria, such as suspicious persons.

[0072] <Second Embodiment> Next, an image processing system 1 comprising an image processing apparatus 100 according to a second embodiment of the present invention will be described. The image processing system 1 according to this embodiment, similar to the image processing system 1 shown in Figure 2, comprises a center system 10, an input device 20, a first edge device 30a, a second edge device 30b, a third edge device 30c, a first camera device 40a, a second camera device 40b, and a third camera device 40c. The difference between the image processing system 1 according to this embodiment and the image processing system 1 according to the first embodiment is the input device 20.

[0073] Figure 14 shows the configuration of the input device 20 according to this embodiment. Next, the configuration of the input device 20 according to this embodiment will be described. As shown in Figure 14, the input device 20 according to this embodiment further includes a location identification unit 204 and a camera selection information generation unit 205, in addition to the communication unit 201, information input unit 202, and display control unit 203 shown in Figure 4.

[0074] The location identification unit 204 identifies the location of the input device 20. The communication unit 201 transmits and receives information with the center system 10. The camera selection information generation unit 205 generates camera selection information that identifies the camera that captures the appearance of a person in the captured video, based on the location of the input device 20 identified by the location identification unit 204. For example, the camera selection information generation unit 205 generates camera selection information that selects a camera that meets predetermined conditions, such as a camera installed within 1 kilometer of the location of the input device 20 identified by the location identification unit 204, or the camera installed closest to the location of the input device 20.

[0075] Next, the processing of the image processing system 1 according to this embodiment will be described. This section describes the process when the image processing system 1 monitors a stalker. In this embodiment, the person being stalked is the user. The processing flow of the image processing system 1 according to this embodiment is the same as that of the image processing system 1 according to the first embodiment shown in Figures 12 and 13, except that the processing of steps S1 and S3 is different.

[0076] Without performing the process of inputting camera selection information in step S1, the information input unit 202 determines whether or not to start face extraction in step S2. If the information input unit 202 does not detect a start instruction input operation to input a start instruction to start face extraction, it returns to the process of step S2. If the information input unit 202 detects a start instruction input operation, instead of the process of step S3, the location identification unit 204 identifies the location of the input device 20 at predetermined time intervals while the user carrying the input device 20 is moving. For example, the location identification unit 204 identifies the location of the input device 20 using the function of GPS (Global Positioning System). The location identification unit 204 outputs the identified location of the input device 20 to the camera selection information generation unit 205 at predetermined time intervals. The camera selection information generation unit 205 generates camera selection information based on the location of the input device 20 input from the location identification unit 204. The camera selection information generation unit 205 transmits the generated camera selection information to the center system 10.

[0077] Each time the camera selection unit 105 determines that it has received camera selection information in step S4, it should perform the processing from step S5 onward.

[0078] Furthermore, if the storage unit 106 of the image processing device 100 stores predetermined conditions such as cameras installed within 1 kilometer of the input device 20's location, or the camera closest to the input device 20's location, the input device 20 may not have a camera selection information generation unit 205 and may transmit the location of the input device 20 identified by the location identification unit 204. Alternatively, the center system 10 may directly acquire the location of the input device 20 from GPS. In these cases, the camera selection unit 105 of the image processing device 100 only needs to identify the edge device based on the acquired location of the input device 20 and the predetermined conditions stored in the storage unit 106.

[0079] The image processing system 1 according to the second embodiment of the present invention has been described above. In the image processing device 100 of the image processing system 1 described above, the appearance state determination unit 101 determines the appearance state of a person in the captured video based on the person candidate determination conditions. The predetermined person candidate estimation unit 102 estimates a predetermined person candidate based on the appearance state determined by the appearance state determination unit 101. This allows for the identification of individuals who meet specified criteria, such as suspicious persons.

[0080] Furthermore, the position identification unit 204 identifies the position of the input device 20 at predetermined time intervals while the user carrying the input device 20 is moving. The position identification unit 204 outputs the identified position of the input device 20 to the camera selection information generation unit 205 at predetermined time intervals. The camera selection information generation unit 205 generates camera selection information based on the position of the input device 20 input from the position identification unit 204. The camera selection information generation unit 205 transmits the generated camera selection information to the central system 10. The camera selection unit 105 performs the processing from step S5 onward each time it determines that it has received the camera selection information generated by the camera selection information generation unit 205. This allows for situations where the camera used to capture the appearance of a person in the footage, such as when monitoring a stalker, needs to be changed moment by moment.

[0081] In the embodiments of the present invention, as long as information is transmitted and received appropriately, some or all of the functional units of the image processing device 100 may be provided in any of the first edge device 30a, second edge device 30b, third edge device 30c, ... nth edge device 30n. Alternatively, some or all of the functional units of the image processing device 100 may be distributed and provided in any of the first edge device 30a, second edge device 30b, third edge device 30c, ... nth edge device 30n.

[0082] In the embodiments of the present invention, as long as information is transmitted and received appropriately, some or all of the functional units of the first edge device 30a, the second edge device 30b, the third edge device 30c, ... the nth edge device 30n may be provided in the center system 10.

[0083] In an embodiment of the present invention, if the input device 20 is to be installed at any fixed location (for example, if the position of the input device 20 is unrelated to the position of the camera taking the picture, as in the first embodiment), then some or all of the functional parts of the input device 20 may be provided in the center system 10, as long as the transmission and reception of information is performed appropriately. Also, in an embodiment of the present invention, if the input device 20 is to be installed at any fixed location, then some or all of the functional parts of the input device 20 may be provided in any of the first edge device 30a, second edge device 30b, third edge device 30c, ... nth edge device 30n, as long as the transmission and reception of information is performed appropriately.

[0084] In the embodiments of the present invention, the image processing device 100 may be equipped with one or more of the first feature extraction unit 305a, the second feature extraction unit 305b, the third feature extraction unit 305c, ... the nth feature extraction unit 305n, as long as information is transmitted and received appropriately.

[0085] Furthermore, each of the storage unit 106, the first storage unit 304a, the second storage unit 304b, and the third storage unit 304c in this invention may be located anywhere within the range where appropriate information can be transmitted and received. Also, each of the storage units 106, the first storage unit 304a, the second storage unit 304b, and the third storage unit 304c may exist in multiple units within the range where appropriate information can be transmitted and received, and data may be stored in a distributed manner.

[0086] In the embodiments of the present invention, the order of processing may be changed as long as appropriate processing is performed.

[0087] Although embodiments of the present invention have been described, each of the image processing apparatus 100, input device 20, first edge device 30a, second edge device 30b, and third edge device 30c described above has a computer system inside. The process described above is stored in a computer-readable storage unit in the form of a program, and the above process is performed when the computer reads and executes this program. Here, a computer-readable storage unit refers to a magnetic disk, magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, etc. Alternatively, this computer program may be distributed to a computer via a communication line, and the computer that receives this distribution may execute the program.

[0088] Furthermore, the above program may be intended to implement some of the functions described above. It may also be a so-called differential file (differential program) that can implement the aforementioned functions in combination with programs already recorded in the computer system.

[0089] Although several embodiments of the present invention have been described, these embodiments are presented as examples only and do not limit the scope of the invention. Furthermore, various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. [Explanation of Symbols]

[0090] 1. Image processing system 10. Center System 20.. Input device 30a...First edge device 30b...Second edge device 30c...Third edge device 40a...First camera device 40b...Second camera device 40c...Third camera device 100...Image processing device 101... Appearance state determination unit 102...Predetermined person candidate estimation unit 103, 201... Communications Department 104... Control Unit 105...Camera Selection Section 106...Storage section 107...Condition Input Section 202... Information Input Section 203...Display Control Unit 204...Location specifying section 205...Camera selection information generation unit 301a...First Communication Department 301b...Second Communication Department 301c...Third Communication Department 302a...First Control Unit 302b...Second Control Unit 302c...Third Control Unit 303a...First Camera Control Unit 303b...Second Camera Control Unit 303c...Third Camera Control Unit 304a...First storage section 304b...Second storage section 304c...Third memory section 305a...First feature extraction unit 305b...Second feature extraction unit 305c...Third Feature Extraction Unit 306a...First video receiving unit 306b...Second video receiving unit 306c...Third video receiving unit 307a...First camera interface 307b...Second camera interface 307c...Third camera interface

Claims

1. A first transmitting unit of an edge device that transmits person information, which is information about a person appearing in a video captured by a camera selected from among multiple cameras installed in multiple stores, and the video to a central system, The receiving unit of the center system receives the person information and the video from the first transmitting unit, The storage unit of the center system stores the person information and video received by the receiving unit, and the judgment conditions. When the person performs a predetermined action, the determination unit of the center system determines that the person is a predetermined person based on the determination conditions stored in the memory unit, The receiving unit receives the person information and the video, and when the determination unit determines that the person is a predetermined person, the second transmitting unit of the center system transmits predetermined information to an external device. An information processing system equipped with the following features.

2. The aforementioned person information consists of person features, which are information used to identify the person. The information processing system according to claim 1.

3. The aforementioned predetermined information is data that reveals the appearance of the person who has been identified as the aforementioned predetermined person. The information processing system according to claim 1 or 2.

4. The information processing system according to any one of claims 1 to 3, wherein the predetermined information is data captured by the selected camera.

5. The edge device transmits to a central system person information, which is information about a person appearing in a video captured by a camera selected from among multiple cameras installed in multiple stores, and the video itself. The central system receives the person information and the video from the edge device, The aforementioned center system stores the received person information and video, and the determination conditions. The center system determines, based on the determination conditions, that the person is a predetermined person when the person performs a predetermined action. The aforementioned center system receives the person information and the video, and when it determines that the person is the predetermined person, it transmits predetermined information to an external device. A processing method that includes this.

Images