PROCESSING SYSTEM, COMMUNICATION SYSTEM, AND PROCESSING METHOD

Abstract

To provide a processing system, a communication system and a processing method for detecting a specific object even when being divided into a plurality of areas by a single imaging unit.SOLUTION: In a distributed processing system, image data representing a predetermined object is transmitted to a centralized data processing server 7. An object ID for identifying a granted object is received from the centralized data processing server 7 as a result of collation by the centralized data processing server 7. A first region number for identifying a first region among a plurality of regions included in a region imaged by a single imaging unit and the object ID of the object existing in the first region are associated with each other and transmitted to the centralized data processing server 7 (S304).SELECTED DRAWING: Figure 41

Description

[Technical field]

[0001] The present disclosure relates to a processing system, a communication system, and a processing method. [Background technology]

[0002] To provide effective promotional information selected based on people's actions in real time The aim is to use multiple cameras to analyze the behavior of people and other objects in a wide range of video footage and A marketing information providing system that generates motion trajectory information is disclosed (see Patent Document 1). see). Summary of the Invention [Problem to be solved by the invention]

[0003] However, in the past, it was possible to capture the movement of an object within a specific area, but it was not known to divide the area into multiple areas with a single imaging unit. Therefore, when the area was divided into multiple areas, a problem arose in which it was not possible to detect a specific object. [Means for solving the problem]

[0004] The invention according to claim 1 includes a transmitting means for transmitting image data representing a predetermined object to a management device that performs image matching, and a receiving means for receiving object identification information for identifying the object that is assigned as a result of the matching by the management device from the management device. With a single imaging unit Images obtained by taking pictures Among the several areas included in A predetermined area image is an image of a predetermined area. Multiple at the same time A display control means for displaying the Multiple Predefined area image Each of a reception means for receiving a setting of a detectable range in which an object can be detected in the captured image, the detectable range including a matchable range in which an image of the object can be matched, the matchable range being a predetermined area within the detectable range, and the transmission means The aboveA processing system characterized by associating first area identification information for identifying a first area among a plurality of areas with the object identification information of the object present in the first area and transmitting the information to the management device. Effect of the Invention

[0005] As described above, according to the present invention, an image is divided into a plurality of regions by a single imaging unit. This provides the effect of making it possible to detect a specific object even when the object is in a confined space. [Brief description of the drawings]

[0006] [Figure 1] 1 is a schematic diagram of a communication system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a hardware configuration diagram of a real-time data processing terminal. [Diagram 3] FIG. 2 is a hardware configuration diagram of an imaging unit. [Figure 4] Figure 4(a) shows a hemispherical image (front side) captured by imaging unit 40b, Figure 4(b) shows a hemispherical image (rear side) captured by imaging unit 40b, and Figure 4(c) shows an image represented using equirectangular projection. [Diagram 5] (a) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (b) is a diagram showing a spherical image. [Figure 6] FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is treated as a three-dimensional solid sphere. [Figure 7] FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 8] 13 is a diagram showing the relationship between predetermined region information and an image of a predetermined region T. FIG. [Figure 9] FIG. 2 is a hardware configuration diagram of a near-terminal data processing device or a distributed data processing terminal. [Figure 10] FIG. 2 is a hardware configuration diagram of a centralized data processing server. [Figure 11]FIG. 2 is a software configuration diagram of a real-time data processing terminal and a near-terminal data processing device in the image acquisition terminal. [Figure 12] FIG. 1 is a functional block diagram of a communication system. [Figure 13] FIG. 1 is a functional block diagram of a communication system. [Figure 14] 1A is a conceptual diagram of an imaging element information management table, and FIG. 1B is a conceptual diagram of a cycle value management table. [Figure 15] 1A is a conceptual diagram of an image acquisition program management table, FIG. 1B is a conceptual diagram of a synthesis processing program management table, FIG. 1C is a conceptual diagram of a distortion correction program management table, and FIG. [Figure 16] FIG. 13 is a conceptual diagram of an object information management table. [Figure 17] FIG. 13 is a conceptual diagram of an object displacement management table. [Figure 18] 1A is a conceptual diagram of an object presence management table, and FIG. 1B is a conceptual diagram of a display area management table. [Figure 19] 1A is a conceptual diagram of an object management table, FIG. 1B is a conceptual diagram of an object position management table, and FIG. 1C is a conceptual diagram of a count management table. [Figure 20] FIG. 2 is a conceptual diagram of a session management table. [Figure 21] FIG. 1 is a conceptual diagram of a terminal ID. [Figure 22] FIG. 4 is a conceptual diagram of an authentication server management table. [Diagram 23] FIG. 4 is a conceptual diagram of an authentication management table. [Figure 24] FIG. 11 is a sequence diagram showing an authentication process. [Diagram 25] FIG. 11 is a sequence diagram showing an authentication process. [Figure 26] FIG. 13 is a diagram showing an example of a screen of a distributed data processing terminal. [Figure 27] FIG. 13 is a diagram showing an example of a screen of a distributed data processing terminal. [Figure 28] FIG. 11 is a sequence diagram showing a process of setting a detection area. [Figure 29]FIG. 11 is a sequence diagram showing a process of setting a detection area. [Diagram 30] FIG. 13 is a diagram showing a display area setting screen. [Diagram 31] FIG. 13 is a diagram showing the positions of detection areas in an actual store. [Diagram 32] FIG. 13 is a diagram showing a detection range setting screen. [Diagram 33] FIG. 33(a) is a diagram showing a predetermined area image Q11 before the change in the detectable range P, FIG. 33(b) is a diagram showing a predetermined area image Q111 after the change in the detectable range P, and FIG. 33(c) is a diagram showing an example of a warning display for a predetermined area image Q112 after the change in the detectable range P. [Diagram 34] FIG. 11 is a sequence diagram showing processing of a request to start image recognition. [Diagram 35] FIG. 11 is a sequence diagram showing preparation processing for real-time processing of the real-time data processing terminal. [Diagram 36] FIG. 11 is a sequence diagram showing a program acquisition process. [Figure 37] FIG. 11 is a sequence diagram showing an image recognition process. [Figure 38] 11 is a flowchart showing object detection processing in real-time processing. [Figure 39] 11 is a flowchart showing a process of generating an event in real-time processing. [Diagram 40] 11 is a flowchart showing a process of generating an event in real-time processing. [Diagram 41] FIG. 13 is a sequence diagram showing the process of displaying the entry / exit count numbers and storing movement information. [Diagram 42] 13 is a flowchart showing a change display process in the distributed data processing terminal. [Diagram 43] FIG. 11 is a sequence diagram showing a matching process. [Diagram 44] 13 is a flowchart showing a collation process. [Diagram 45] FIG. 13 is a diagram showing a matching result display screen. [Diagram 46] FIG. 11 is a sequence diagram showing a process for displaying movement information. [Figure 47] 13 is a flowchart showing a process for counting the number of people entering and exiting a building entrance, taking into account the moving direction of an object. [Figure 48] FIG. 13 is a diagram showing a movement result display screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] [Overall Overview] Hereinafter, an overall outline of one embodiment of the present invention will be described with reference to the drawings. This embodiment discloses a communication system that realizes edge computing. That is, this embodiment discloses an invention that realizes a service provided by the service providing server 2 at the image acquisition terminal 2 through cooperation between processing at the service providing server 8 and processing at the image acquisition terminal 2.

[0008] [Overall configuration of the embodiment] Fig. 1 is a schematic diagram of a communication system according to this embodiment. As shown in Fig. 1, the communication system according to this embodiment is constructed by a real-time data processing terminal 3, a near-terminal data processing device 5, a distributed data processing terminal 6, a centralized data processing server 7, a service providing server 8, and authentication servers 9a, 9b, and 9c. The real-time data processing terminal 3, the near-terminal data processing device 5, and the distributed data processing terminal 6 construct a distributed processing system 100.

[0009] The near-terminal data processing device 5 is communicatively connected to a distributed data processing terminal 6 via an intranet 200. The distributed data processing terminal 6 is communicatively connected to a centralized data processing server 7, a service providing server 8, and authentication servers 9a, 9b, and 9c via the Internet 600. Note that the authentication server 9 is a collective term for the authentication servers 9a, 9b, and 9c.

[0010] Among these, the real-time data processing terminal 3 is a terminal that performs real-time processing to obtain captured image data. An imaging unit 40 having an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor for capturing an image of a subject is detachably connected to the real-time data processing terminal 3. As a result, the real-time data processing terminal 3 digitizes the captured image data (hereinafter referred to as "captured image data") input from the imaging unit 40, and detects an image of an object (here, a face) in real time (for example, every 1 / 60 seconds). The real-time data processing terminal 3 transmits data of a partial image (hereinafter referred to as "partial image data"), which is a region of the object part in the captured image, to the near-terminal data processing device 5.

[0011] The near-terminal data processing device 5 is disposed closest to the real-time data processing terminal 3, and is closely connected one-to-one with the real-time data processing terminal 3 via a data bus or a USB (Universal Serial Bus), for example, in terms of hardware. The near-terminal data processing device 5 encodes the partial image data received from the real-time data processing terminal 3 into a general-purpose format such as JPEG (Joint Photographic Experts Group), and then transmits the encoded data to the distributed data processing terminal 6 via the intranet 200 as data to be matched when face image matching is performed. Usually, the real-time data processing terminal 3 and the near-terminal data processing device 5 are connected together, and here, the real-time data processing terminal 3 and the near-terminal data processing device 5 constitute the image acquisition terminal 2.

[0012] The distributed data processing terminal 6 is a computer that is used by a user at a location relatively close to the near-terminal data processing device 5 and can accept various operations from the user. The distributed data processing terminal 6 registers and stores in advance matching data when face image matching is performed. The distributed data processing terminal 6 can request the centralized data processing server 7 to match the matching data with the data to be matched via the Internet 600. In this case, the distributed data processing terminal 6 also transmits the data to be matched received from the near-terminal data processing device 5 and the matching data registered in advance. Furthermore, the distributed data processing terminal 6 can receive matching result information indicating the matching result from the centralized data processing server 7 as a response. Moreover, the distributed data processing terminal 6 can display the matching result via a graphic interface.

[0013] The centralized data processing server 7 is installed at a location relatively far from the near-terminal data processing device 5, and can communicate with the distributed data processing terminals 6 via a communication network such as the Internet 600. When the centralized data processing server 7 receives matching request information indicating a matching request, as well as matching data and matched data, it performs matching of the matching data and the matched data and determines the degree of similarity. Then, the centralized data processing server 7 transmits matching result information indicating the matching result including the degree of similarity to the distributed data processing terminals 6.

[0014] The service providing server 8 is a server that provides various services to the image acquisition terminal 2.

[0015] The authentication server 9a performs ID authentication to determine whether the image acquisition terminal 2 has a valid right to receive a service from the service providing system 8. The authentication servers 9b and 9c perform the same authentication.

[0016] [Hardware configuration] Next, the hardware configuration of the communication system of this embodiment will be described with reference to FIGS.

[0017] <Hardware configuration of real-time data processing terminal> 2 is a diagram showing the hardware configuration of the real-time data processing terminal 3. The real-time data processing terminal 3 includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a CMOS sensor 305, an acceleration / direction sensor 306, a media I / F 308, and a GPS receiver 309.

[0018] Among these, the CPU 301 controls the overall operation of the real-time data processing terminal 3. The ROM 302 stores programs used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The EEPROM 304 reads or writes various data such as programs for the real-time data processing terminal under the control of the CPU 301. The CMOS sensor 305 captures an image of a subject (mainly a blind spot of the photographing unit 40) under the control of the CPU 301 to obtain photographed image data. The acceleration / direction sensor 306 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I / F 308 controls the reading or writing (storage) of data from or to a recording medium 307 such as a flash memory. The GPS receiver 309 receives GPS signals from GPS satellites.

[0019] The real-time data processing terminal 3 also includes an imaging unit I / F 313 , a microphone 314 , a speaker 315 , a sound input / output I / F 316 , a display 317 , an external device connection I / F 318 , and a touch panel 321 .

[0020] Among these, the imaging unit I / F 313 is a circuit that controls the driving of the imaging unit 40 when an external imaging unit 40 is connected. The microphone 314 is a type of built-in sound collecting means that inputs sound. The sound input / output I / F 316 is a circuit that processes input and output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display 317 is a type of display means such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 318 is an interface for connecting various external devices. The touch panel 321 is a type of input means that allows a user to operate the real-time data processing terminal 3 by pressing the display 317.

[0021] The real-time data processing terminal 3 also includes a bus line 310. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 301 shown in FIG.

[0022] <Hardware configuration of the imaging unit> Fig. 3 is a hardware configuration diagram of an imaging unit. In particular, Fig. 3(a) is a hardware configuration diagram of a monocular imaging unit 40a among the imaging units 40. Fig. 3(b) is a hardware configuration diagram of a compound-eye imaging unit 40b among the imaging units 40. Note that imaging unit 40 is a collective term for multiple types of imaging units (imaging units 40a, 40b, etc.) that differ in the number of imaging elements.

[0023] 3(a), the imaging unit 40a has an imaging element 401a such as a CMOS or a CCD, a lens 402a, and a connection I / F 408a for electrically connecting to the imaging unit I / F 313 of the real-time data processing terminal 3. When the imaging unit 40a is connected to the imaging unit I / F 313 of the real-time data processing terminal 3, the imaging element 401a captures an image in response to an imaging control signal sent from the imaging unit I / F 313 via the connection I / F 408a, and transmits captured image data to the imaging unit I / F 313 via the connection I / F 408a.

[0024] 3B, the imaging unit 40b has imaging elements 401b1 and 401b2 such as CMOS or CCD, lenses 402b1 and 402b2, and a connection I / F 408b for electrically connecting to the imaging unit I / F 313 of the real-time data processing terminal 3. The lenses 402b1 and 402b2 are, for example, fisheye lenses. When the imaging unit 40b is connected to the imaging unit I / F 313 of the real-time data processing terminal 3, the imaging elements 401b1 and 401b2 capture images according to an imaging control signal sent from the imaging unit I / F 313 via the connection I / F 408b, and transmit a plurality of captured image data to the imaging unit I / F 313 via the connection I / F 408b.

[0025] The imaging unit 40a shown in Fig. 3(a) can obtain a general planar image, while the imaging unit 40b shown in Fig. 3(b) can obtain a celestial sphere image. Here, an outline of the process until an equirectangular projection image EC and a celestial sphere image CE are generated from an image captured by the imaging unit 40b will be described with reference to Figs. 4 to 8. Fig. 4(a) is a diagram showing a hemispherical image (front side) captured by the imaging unit 40b, Fig. 4(b) is a diagram showing a hemispherical image (rear side) captured by the imaging unit 40b, and Fig. 4(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 5(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 5(b) is a diagram showing a celestial sphere image.

[0026] As shown in Fig. 4(a), the image obtained by the image sensor 401b1 becomes a hemispherical image (front side) curved by the lens 402b1. Also, as shown in Fig. 4(b), the image obtained by the image sensor 403b becomes a hemispherical image (rear side) curved by the lens 402b2. Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are synthesized to generate an equirectangular projection image EC as shown in Fig. 4(c).

[0027] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted to cover the sphere as shown in FIG. 5(a), and a celestial sphere image CE as shown in FIG. 5(b) is created. In this manner, the celestial sphere image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere image CE may be a still image or a video.

[0028] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to humans. Therefore, by displaying a predetermined area (hereinafter, referred to as a "predetermined area image") of the spherical image CE as a planar image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to humans. This will be described with reference to Figs. 6 and 7.

[0029] FIG. 6 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user who views the omnidirectional image CE displayed as a three-dimensional sphere. FIG. 7(a) is a three-dimensional perspective view of FIG. 6, and FIG. 7(b) is a diagram showing a predetermined area image when displayed on a display. In FIG. 7(a), the omnidirectional image CE shown in FIG. 5 is represented by a three-dimensional sphere CS. If the omnidirectional image CE generated in this way is a sphere CS, the virtual camera IC is located inside the omnidirectional image CE as shown in FIG. 6. The predetermined area T in the omnidirectional image CE is a shooting area of ​​the virtual camera IC, and is specified by predetermined area information indicating the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE.

[0030] Then, the predetermined area image Q shown in Fig. 7(a) is displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC as shown in Fig. 7(b). The image shown in Fig. 7(b) is a predetermined area image represented by the initial setting (default) predetermined area information. The following description will be given using the shooting direction (ea, aa) and the angle of view (α) of the virtual camera IC.

[0031] The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 8. FIG. 8 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 8, "ea" indicates an elevation angle, "aa" indicates an azimuth angle, and "α" indicates an angle of view. That is, the posture of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP of the predetermined area T, which is the shooting area of ​​the virtual camera IC. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). In FIG. 8, the trigonometric function generally shown in the following (Equation 1) holds. The center point CP can be specified by the shooting direction (ea, aa) and the distance f, so that the predetermined area information can be expressed by the center point CP and the angle of view α.

[0032]

number

[0033] <Hardware configuration of near-terminal data processing device and distributed data processing terminal> 9 is a hardware configuration diagram of the near-terminal data processing device or the distributed data processing terminal. Here, the near-terminal data processing device 5 and the distributed data processing terminal 6 are shown to have the same hardware configuration, so the near-terminal data processing device 5 will be described and the description of the distributed data processing terminal 6 will be omitted.

[0034] As shown in FIG. 9, the near terminal data processing device 5 includes a CPU 501 , a ROM 502 , a RAM 503 , an EEPROM 504 , a CMOS sensor 505 , an acceleration / direction sensor 506 , a media I / F 508 , and a GPS receiving unit 509 .

[0035] Among these, the CPU 501 controls the operation of the entire near-terminal data processing device 5. The ROM 502 stores a program used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as programs for the near-terminal data processing device under the control of the CPU 501. The CMOS sensor 505 captures an object (mainly a self-portrait of the user who operates the near-terminal data processing device 5) under the control of the CPU 501 to obtain captured image data. The acceleration / direction sensor 506 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I / F 508 controls the reading or writing (storage) of data from or to a recording medium 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

[0036] The near-terminal data processing device 5 also includes a long-distance communication circuit 511, an antenna 511a of the long-distance communication circuit 511, a camera 512, an image sensor I / F 513, a microphone 514, a speaker 515, an audio input / output I / F 516, a display 517, an external device connection I / F 518, a short-distance communication circuit 519, an antenna 519a of the short-distance communication circuit 519, and a touch panel 521.

[0037] Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via the intranet 200. The camera 512 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 501 to obtain captured image data. The image sensor I / F 513 is a circuit that controls the driving of the camera 512. The microphone 514 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 516 is a circuit that processes input and output of sound signals between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a type of display means such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 518 is an interface for connecting various external devices. The short-distance communication circuit 519 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 521 is a type of input means that a user operates the near-terminal data processing device 5 by pressing the display 517.

[0038] Further, the near-terminal data processing device 5 includes a bus line 510. The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 501 shown in FIG.

[0039] <Hardware configuration of centralized data processing server, service providing server, and authentication server> 10 is a hardware configuration diagram of the centralized data processing server, the service providing server, or the authentication server. Here, the centralized data processing server 7, the service providing server 8, and the authentication server 9 are shown to have the same hardware configuration, so only the centralized data processing server 7 will be described, and descriptions of the service providing server 8 and the authentication server 9 will be omitted.

[0040] Fig. 10 is a hardware configuration diagram of the centralized data processing server. The centralized data processing server 7 is constructed by a computer, and as shown in Fig. 10, includes a CPU 701, a ROM 702, a RAM 703, an HD 704, an HDD (Hard Disk Drive) 705, a recording medium 706, a media I / F 707, a display 708, a network I / F 709, a keyboard 711, a mouse 712, a DVD-RW drive 714, and a bus line 710. Since the centralized data processing server 7 functions as a server, it does not need to have input devices such as the keyboard 711 and the mouse 712, or output devices such as the display 708.

[0041] Of these, the CPU 701 controls the overall operation of the centralized data processing server 7. The ROM 702 stores programs used to drive the CPU 701. The RAM 703 is used as a work area for the CPU 701. The HD 704 stores various data such as programs. The HDD 705 controls reading or writing of various data from the HD 704 under the control of the CPU 701. The media I / F 707 controls reading or writing (storing) of data from a recording medium 706 such as a flash memory. The display 708 displays various information such as a cursor, a menu, a window, characters, or an image. The network I / F 709 is an interface for data communication using the Internet 600. The keyboard 711 is a type of input means having multiple keys for inputting characters, numerical values, various instructions, and the like. The mouse 712 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, and the like. The DVD-RW drive 714 controls the reading of various data from a DVD-RW (Digital Versatile Disc-ReWritable) 713, which is an example of a removable recording medium.

[0042] 10. The centralized data processing server 7 also includes a bus line 710. The bus line 710 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 701 shown in FIG.

[0043] [Software configuration] FIG. 11 is a software configuration diagram of the real-time data processing terminal and the near-terminal data processing device in the image acquisition terminal.

[0044] 11, the real-time data processing terminal 3 has an OS 300 and an image recognition application (hereinafter, "application" will be referred to as "app") AP1. The image recognition application AP1 runs on the RAM 303 of the real-time data processing terminal 3. Of these, the OS 300 is basic software that provides basic functions and manages the entire real-time data processing terminal 3. The image recognition application AP1 is an application for recognizing the faces of people, animals, etc. from captured images.

[0045] Further, the near terminal data processing device 5 has an OS 500 and a communication application AP2. The communication application AP2 runs on the RAM 503 of the near terminal data processing device 5. Of these, the OS 500 is basic software that provides basic functions and manages the entire near terminal data processing device 5. The communication application AP2 is an application for communicating with other terminals (devices) such as the distributed data processing terminal 6.

[0046] In this way, in the image acquisition terminal 2, the real-time data processing terminal 3 performs image recognition, while the near-terminal data processing device 5 communicates with the distributed data processing terminal 6 via the intranet 200, thereby enabling distributed processing to be performed. The distributed processing system 100 including the image acquisition terminal 2, the intranet 200, and the distributed data processing terminal 6 is an example of a processing system.

[0047] Furthermore, the real-time data processing terminal 3 and the near-terminal data processing device 5 are equipped with not only an OS but also a driver, a software development kit (SDK), or an application programming interface (API), which may be different from each other.

[0048] Next, the functional configuration and processing of an embodiment of the present invention will be described with reference to the drawings.

[0049] [Functional configuration of the embodiment] First, the functional configurations of the terminals, devices, and servers constituting the communication system according to the present embodiment will be described with reference to Fig. 12 to Fig. 23. Fig. 12 is a functional block diagram of the communication system according to the embodiment. Fig. 12 particularly shows the functional blocks of the image acquisition terminal 2.

[0050] <Functional configuration of real-time data processing terminal> 12, the real-time data processing terminal 3 has a calculation unit 31, a judgment unit 33, an image processing unit 34, an object detection unit 35, an event generation unit 36, a display control unit 37, a connection unit 38, a storage / readout unit 39, and a communication unit 48. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 2 by an instruction from the CPU 301 in accordance with the program loaded from the EEPROM 304 onto the RAM 303.

[0051] 2. Shape model data, which will be described later, is stored in the storage unit 3000. Furthermore, an image sensor information management DB 3001, a cycle value management DB 3002, an image acquisition program management DB 3003, a synthesis processing program management DB 3004, a distortion correction program management DB 3005, a service program management DB 3006, an object information management DB 3007, and an object displacement management DB 3008 are stored in the storage unit 3000.

[0052] The imaging element information management DB 3001 is made up of an imaging element information management table, which will be described later. The cycle value management DB 3002 is made up of a cycle value management table, which will be described later. The image acquisition program management DB 3003 is made up of an image acquisition program management table, which will be described later. The synthesis processing program management DB 3004 is made up of a synthesis processing program management table, which will be described later. The distortion correction program management DB 3005 is made up of a distortion correction program management table, which will be described later. The service program management DB 3006 is made up of a service program management table, which will be described later.

[0053] Furthermore, the object information management DB 3007 is configured by an object information management table, which will be described later.The object displacement management DB 3008 is configured by an object displacement management table, which will be described later.

[0054] (Image sensor information management table) 14(a) is a conceptual diagram showing an imaging element information management table. In this imaging element information management table, the model number of the imaging unit 40, the number of imaging elements included in the imaging unit 40, and the type of lens used in the imaging elements are managed in association with each other. The model number is an example of type information that indicates the type of imaging unit according to the number of imaging elements and the type of lens used in the imaging elements.

[0055] (Cycle value management table) 14B is a conceptual diagram showing a cycle value management table, in which the number of imaging elements included in the imaging unit 40 and a cycle value (frames per second) of object recognition processing, which will be described later, are associated and managed.

[0056] Each table shown in FIG. 15 is a table for managing whether or not each program is installed in the real-time data processing terminal 3.

[0057] (Image acquisition program management table) 15(a) is a conceptual diagram showing an image acquisition program management table. In this image acquisition program management table, the number of imaging elements that the imaging unit 40 has and the image acquisition program (or the name thereof) are managed in association with each other. For example, when an image acquisition program for the number of imaging elements "1" is installed in the real-time data processing terminal 3, the program "ProgC01 (image acquisition of one system)" is managed in association with each other. When an image acquisition program for the number of imaging elements "2" is installed in the real-time data processing terminal 3, the program "ProgC02 (image acquisition of two systems)" is managed in association with each other.

[0058] (Synthesis processing program management table) 15B is a conceptual diagram showing a compositing process program management table, which manages the number of image pickup elements included in the imaging unit 40 and the names of compositing processes in association with each other.

[0059] (Distortion correction program management table) 15(c) is a conceptual diagram showing a distortion correction program management table. In this distortion correction program management table, the type of lens that the imaging unit 40 has and the distortion correction program (or the name thereof) are managed in association with each other. For example, when a distortion correction program for the lens type "wide angle" is installed in the real-time data processing terminal 3, the program "ProgW01 (wide-angle distortion correction)" is managed in association with each other. When a distortion correction program for the lens type "fisheye" is installed in the real-time data processing terminal 3, the program "ProgW02 (fisheye distortion correction)" is managed in association with each other.

[0060] (Service program management table) 15(d) is a conceptual diagram showing a service program management table. In this service program management table, an authentication server ID for identifying an authentication server and the name of a service program for executing a service provided by the image acquisition terminal 2 are managed in association with each other. For example, when a service program for authentication server ID "a01" is installed in the real-time data processing terminal 3, a program "ProgD01 (object detection)" is managed in association with each other. Also, when a service program for authentication server ID "a02" is installed in the real-time data processing terminal 3, a program "ProgD02 (object counting)" is managed in association with each other.

[0061] (Object information management table) 16 is a conceptual diagram showing an object information management table. In this object information management table, a (serial number), the position of a rectangular partial image in a detectable range P described below, the width of the partial image, the height of the partial image, and an end flag (here, "*") are managed in association with each other. The position of the partial image indicates the coordinate position of the upper left corner of the rectangular image.

[0062] (Object displacement management table) 17 is a conceptual diagram showing an object displacement management table. In this object displacement management table, the image ID of a partial image for identifying a partial image, the position (coordinates) of this partial image in the collation possible range C, the width of the partial image, the height of the partial image, the collation progress status indicating the progress of the collation of this partial image in the centralized data processing server 7, the object ID for identifying an object (here, a person) shown in the partial image, and the end flag (here, "*") are managed in association with each other. Among these, the position of the partial image in the detectable range P, the width of the partial image, and the height of the partial image are object identification information transferred from the object information management table (see FIG. 16). That is, the information (position, width, height) in the object information management table is partial image identification information for identifying the current partial image, and the information (position, width, height) in the object displacement management table is partial image identification information for identifying the past partial image.

[0063] The object ID is an example of object identification information for identifying an object. Examples of object identification information include employee numbers, student ID numbers, driver's license numbers, and My Numbers used in the Japanese system.

[0064] Furthermore, an "end flag" in the object information management table (see FIG. 16) is added to indicate that the determination is completed when it is determined that the current position of an object in any record in the object information management table is within a predetermined distance (threshold) from the past position of the object in each record of the object displacement management table (see FIG. 17). On the other hand, when there is partial image specifying information transferred from the object information management table, the "end flag" in the object displacement management table is added to the record including this partial image specifying information.

[0065] (Functional configuration of real-time data processing terminal) Next, each functional configuration of the real-time data processing terminal 3 will be described in more detail with reference to FIG.

[0066] The calculation unit 31 of the real-time data processing terminal 3 is realized by the processing of the CPU 301, and calculates the distance between the latest position of the object and the past position of the object.

[0067] The determination unit 33 is realized by the processing of the CPU 301, and performs various determinations. For example, the determination unit 33 determines the number of image pickup elements based on the model number sent from the image pickup unit 40, with reference to the image pickup element information management DB 3001.

[0068] The image processing unit 34 is realized by the processing of the CPU 301, and performs various image processing by executing each program (image acquisition program, image synthesis program, distortion correction program, and service program) shown in Fig. 15. Specifically, the image processing unit 34 performs first image processing (e.g., image acquisition, image synthesis, distortion correction) on image data by executing a first program (e.g., image acquisition program, image synthesis program, and distortion correction program) that does not require authentication to be acquired. Also, the image processing unit 34 performs second image processing (e.g., object detection processing, object counting processing) on ​​image data by executing a second program (e.g., service program) that requires authentication to be acquired.

[0069] The object detection unit 35 is realized by processing of the CPU 301, and detects feature points that are candidates for a predetermined specific object such as a face in the captured image data acquired by the image processing unit 34, and detects the position of the object in the captured image by referring to shape model data that indicates a shape model of the predetermined specific object such as a face.

[0070] Event generation unit 36 ​​is realized by processing of CPU 301, and generates detection information (event information) indicating that object detection unit 35 has detected the position of an object.

[0071] The display control unit 37 is realized by the processing of the CPU 301, and causes a display 317, which is an example of a display means, to display various screens.

[0072] The connection unit 38 is implemented by the processing of the imaging unit I / F 313 and the CPU 301, and is an interface for mechanically and electrically connecting the imaging unit 40 to the real-time data processing terminal 3.

[0073] The storage / reading unit 39 is realized by the processing of the CPU 301 , and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000 .

[0074] The communication unit 48 is realized by the processing of the external device connection I / F 318 and the CPU 301, and transmits and receives various data (or information) to and from a communication unit 58 (described later) of the near-terminal data processing device 5 via one-to-one communication. This communication may be wireless as well as wired.

[0075] <Functional configuration of near-terminal data processing device> 12, the near-terminal data processing device 5 has a transmitting / receiving unit 51, a data detecting unit 56, a display control unit 57, a communication unit 58, and a memory / reading unit 59. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 9 by an instruction from the CPU 501 in accordance with a program loaded from the EEPROM 504 onto the RAM 503 in the near-terminal data processing device 5.

[0076] The near-terminal data processing device 5 also has a storage unit 5000 constructed by a ROM 502, a RAM 503, and an EEPROM 504 shown in FIG.

[0077] (Functional configuration of near-terminal data processing device) Next, each functional configuration of the near terminal data processor 5 will be described in more detail with reference to FIG.

[0078] The transmitter / receiver unit 51 of the near-terminal data processing device 5 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501, and transmits and receives various data (or information) with the distributed data processing terminal 6 via a communication network (here, the intranet 200).

[0079] The data detection unit 56 is realized by the processing of the CPU 501, and detects whether or not an event that requires reception of data from the real-time data processing terminal 3 has occurred, and whether or not reception of the data has been completed.

[0080] A display control unit 57 is realized by the processing of the CPU 501, and causes a display 517, which is an example of a display means, to display various screens.

[0081] The communication unit 58 is realized by the processing of the external device connection I / F 518 and the CPU 501, and transmits and receives various data (or information) to and from the communication unit 48 of the real-time data processing terminal 3 through one-to-one communication. This communication may be wireless or wired.

[0082] The storage / reading unit 59 is realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000 .

[0083] <Functional configuration of distributed data processing terminal> 13, the distributed data processing terminal 6 has a transmitting / receiving unit 61, a receiving unit 62, a determining unit 63, a calculating unit 65, a display control unit 67, and a storing / reading unit 69. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 9 in accordance with an instruction from the CPU 501 in accordance with a program loaded from the EEPROM 504 onto the RAM 503 in the distributed data processing terminal 6.

[0084] Moreover, the distributed data processing terminal 6 has a storage unit 6000 constructed by the ROM 502, the RAM 503, and the EEPROM 504 shown in Fig. 9. In the storage unit 6000, an object presence management DB 6001 and a display area management DB 6002 are constructed.

[0085] The object presence management DB 6001 is configured with an object presence management table, which will be described later. The display area management DB 6002 is configured with a display area management table.

[0086] (Object presence management table) 18(a) is a conceptual diagram showing an object presence management table provided in the object presence management DB 6001. In this object presence management table, object IDs for identifying objects such as people are managed. This makes it possible to manage that an object identified by an object ID is present in any area (areas a1 to a4 shown in FIG. 31 described later) in the store.

[0087] (Display area management table) Fig. 18(b) is a conceptual diagram showing a display area management table provided in the display area management DB 6002. In this display area management table, areas within the store displayed in each of the detection area display fields d11 to d14 shown in Fig. 30 described later are managed.

[0088] (Functional configuration of distributed data processing terminal) The transmitting / receiving unit 61 of the distributed data processing terminal 6 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501 in the distributed data processing terminal 6, and transmits and receives various data (or information) to and from the centralized data processing server 7 via a communication network (here, the Internet 600). For example, the transmitting / receiving unit 61 transmits a request for matching between matching data and matched data to the centralized data processing server 7, and processes the matching result sent from the centralized data processing server 7.

[0089] The reception unit 62 is realized by the processing of the touch panel 521 and the CPU 501 in the distributed data processing terminal 6, and receives various operations from the user.

[0090] The determination unit 63 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and performs various determinations.

[0091] The calculation unit 65 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and performs various calculations.

[0092] The display control unit 67 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and causes various screens to be displayed on the display 517, which is an example of a display means.

[0093] The storage / reading unit 69 is realized by processing of the CPU 501 in the distributed data processing terminal 6, and stores various data (or information) in the storage unit 6000 and reads various data (or information) from the storage unit 6000. For example, in response to a registration request accepted by the acceptance unit 62, the storage / reading unit 69 stores and registers matching data (here, face image data) in the storage unit 6000.

[0094] <Functional configuration of the centralized data processing server> 13, the centralized data processing server 7 has a transmitting / receiving unit 71, a receiving unit 72, a determining unit 73, a feature generating unit 74, a matching unit 75, a counting unit 76, and a storage / reading unit 79. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 10 by an instruction from the CPU 701 in accordance with a program loaded from the HD 704 onto the RAM 703 in the distributed data processing terminal 6.

[0095] The centralized data processing server 7 also has a storage unit 7000 constructed by a ROM 702, a RAM 703, and a HD 704 shown in FIG. 10. The storage unit 7000 stores feature amount data on the matching side, which will be described later. The matching data is stored in this storage unit 7000. Furthermore, various DBs 7001, 7002, and 7003 are constructed in the storage unit 7000. The matching data management DB 7001 is constructed by a matching data management table, which will be described later. Note that the matching data may be stored in a data management server other than the centralized data processing server 7. Furthermore, an object position management DB 7002 is constructed in the storage unit 7000. The object position management DB 7002 is constructed by an object position management table, which will be described later. Furthermore, a counting management DB 7003 is constructed in the storage unit 7000. The counting management DB 7003 is constructed by a counting management table, which will be described later.

[0096] (Matching data management table) 19(a) is a conceptual diagram showing a matching data management table provided in the matching data management DB 7001. In this matching data management table, an image file indicating the file name of the matching data and an object ID for identifying an object such as a person shown in the matching image related to this image file are managed in association with each other.

[0097] (Object position management table) FIG. 19(b) is a conceptual diagram showing an object position management table provided in the object position management DB 7002. In this object position management table, the reception date and time, the area number, and the object ID are managed in association with each other. Of these, the reception date and time indicates the date and time when the centralized data processing server 7 received a movement information storage request from the distributed data processing terminal 6 (see S306). The area number is information indicating the area within the store where the object as a person was present. For example, the area number "1" indicates that the person was present (located) in area a1 within the store shown in FIG. 31. The object ID is the same as the object ID in the matching data management table.

[0098] (Counting management table) 19(c) is a conceptual diagram showing a counting management table provided in the counting management DB 7003. In this counting management table, the number of people or other objects that move to a movement area (left, up, right, down) is associated and managed for each time period (e.g., 1 hour) in the store.

[0099] (Functional configuration of the centralized data processing server) The transmitting / receiving unit 71 of the centralized data processing server 7 is realized by the processing of the network I / F 709 and the CPU 701, and transmits and receives various data (or information) to and from the distributed data processing terminals 6 via a communication network (here, the Internet 600). The transmitting / receiving unit 71 receives a request for matching between matching data and matched data from the distributed data processing terminals 6, and transmits matching result information indicating the matching result to the distributed data processing terminals 6.

[0100] The reception unit 72 is realized by the processing of the keyboard 711, the mouse 712, and the CPU 701, and receives various operations from the user.

[0101] The determination unit 73 is realized by the processing of the CPU 501 in the centralized data processing server 7, and performs various determinations.

[0102] The feature amount generating unit 74 is realized by processing of the CPU 701, and generates feature amount parameters from the matched data (partial image data) and matching data received by the transmitting / receiving unit 71.

[0103] The matching unit 75 is realized by processing of the CPU 701, and uses the features generated by the feature generation unit 74 to match the features of the matching data side with the features of the data to be matched, and calculates a score (points) indicating the degree of similarity.

[0104] The counting unit 76 is realized by processing of the CPU 701, and counts the number of records for each combination of direction and object ID for each unit time (each hour in this embodiment) in the movement information stored in the object position management DB 7002.

[0105] The storage / reading unit 79 is realized by the processing of the CPU 701 , and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000 .

[0106] <Functional configuration of the service provider server> Next, each functional configuration of the service providing server 8 will be described in detail with reference to Fig. 13 and Fig. 20 to Fig. 22. As shown in Fig. 13, the service providing server 8 has a transmitting / receiving unit 81, a determining unit 82, an extracting unit 87, and a storing / reading unit 89. Each of these units is a function or means realized by any of the components shown in Fig. 10 operating in response to an instruction from the CPU 701 in accordance with the program for the service providing server 8 loaded from the HD 704 onto the RAM 703.

[0107] The service providing server 8 also has a storage unit 8000 constructed by the RAM 703 and HD 704 shown in Fig. 10. This storage unit 8000 stores various data sent from the distributed data processing terminal 6 or the authentication server 9. The storage unit 8000 also stores all of the programs shown in Fig. 15, and the service providing server 8 can transmit a requested program in response to a request from the real-time data processing terminal 3.

[0108] Furthermore, the storage unit 8000 has a session management DB 8001 and an authentication server management DB 8002. Of these, the session management DB 8001 is configured with a session management table, which will be described later. The authentication server management DB 8002 is configured with an authentication server management table, which will be described later. Each table will be described in detail below.

[0109] (Session management table) 20 is a conceptual diagram showing a session management table provided in the session management DB 8001. In this session management table, a session ID for identifying a communication session to be established with the distributed data processing terminal 6 in order to provide a service to the distributed data processing terminal 6, a terminal ID for identifying a user of the distributed data processing terminal 6, and an IP address of the distributed data processing terminal 6 of the user identified by the terminal ID are stored and managed in association with each other.

[0110] 21(a), (b), and (c) show an email address as an example of a terminal ID (identification), each of which is composed of an authentication subject portion and a non-authentication subject portion. The authentication subject portion is a user ID used when authenticating by the authentication server 9. The non-authentication subject portion is a portion not used when authenticating by the authentication server 9.

[0111] Of these, in the first pattern shown in Fig. 21(a), the part to be authenticated is composed of the account name "asai", the host name "myhost", and the first part of the domain name "ricoo.com". On the other hand, the part not to be authenticated is composed of the last part of the domain name "theta1". In this case, the extraction unit 87 distinguishes between the part to be authenticated and the part not to be authenticated by using " / ".

[0112] Fig. 21(b) also shows the first pattern, but the non-authentication target part is different from that of Fig. 21(a). That is, the authentication server 9 authenticates the terminal ID shown in Fig. 21(a) as the same as the terminal ID shown in Fig. 21(b) because the authentication target part is the same.

[0113] The terminal ID may be a second pattern shown in Fig. 21(c). In this second pattern, the part to be authenticated is made up of the first part of the account name, "asai". On the other hand, the part not to be authenticated is made up of the last part of the account name, "theta2", the host name, "myhost", and the domain name, "ricoo.com". In this case, the extraction unit 87 distinguishes between the part to be authenticated and the part not to be authenticated by using "+".

[0114] (Authentication server management table) 22 is a conceptual diagram showing an authentication server management table. In this authentication server management table, a URL (Uniform Resource Locator) for accessing each authentication server 9 is stored and managed in association with each authentication server ID for identifying each authentication server 9.

[0115] (Service server functional configuration) Next, each functional configuration of the service providing server 8 will be described in detail with reference to FIG.

[0116] The transmission / reception unit 81 of the service providing server 8 is mainly realized by instructions from the CPU 701 shown in Figure 10 and the network I / F 709, and transmits and receives various data (or information) with the distributed data processing terminal 6 or the authentication server 9 via the Internet 600.

[0117] The determination unit 82 is mainly realized by an instruction from the CPU 701 shown in FIG. 10, and determines, for example, whether or not a communication session for providing a service to the distributed data processing terminal 6 has already been established.

[0118] The extracting unit 87 is mainly realized by commands from the CPU 701 shown in FIG. 10, and performs processing to extract a user ID (part to be authenticated) as shown in FIG. 21 from the terminal ID.

[0119] The storage / reading unit 89 is realized mainly by instructions from the CPU 701 shown in FIG. 10 and the HDD 705 , and stores various data in the storage unit 8000 and reads various data from the storage unit 8000 .

[0120] <Authentication server functional configuration> Next, the functional configuration of the authentication server 9 will be described in detail with reference to Fig. 13 and Fig. 23. The authentication server 9 has a transmission / reception unit 91, an authentication unit 92, and a storage / readout unit 99. Each of these units is a function or means realized by any of the components shown in Fig. 10 operating in response to an instruction from the CPU 701 in accordance with the program for the authentication server 9 loaded from the HD 704 onto the RAM 703.

[0121] 10. The authentication server 9 also has a storage unit 9000 constructed by the RAM 703 and the HD 704 shown in Fig. 10. In this storage unit 9000, various data transmitted from the distributed data processing terminal 6 or the service providing server 8 is stored.

[0122] Furthermore, an authentication management DB 9001 is constructed in the storage unit 9000. The authentication management DB 9001 is composed of an authentication management table, which will be described later. This table will be described in detail below.

[0123] (Authentication Management Table) Figure 23(a) is a conceptual diagram showing an authentication management table held by authentication server 9a, Figure 23(b) is a conceptual diagram showing an authentication management table held by authentication server 9b, and Figure 23(c) is a conceptual diagram showing an authentication management table held by authentication server 9c.

[0124] In each authentication management table, a user ID (part to be authenticated) of a terminal ID and a password are stored and managed in association with each other.

[0125] (Authentication server functional configuration) Next, each functional configuration of the authentication server 9 will be described in detail with reference to FIG.

[0126] The transmission / reception unit 91 of the authentication server 9 is mainly realized by instructions from the CPU 701 shown in FIG. 10 and the network I / F 709, and transmits and receives various data (or information) with the distributed data processing terminal 6 and the service providing server 8 via the Internet 600.

[0127] The authentication unit 92 is mainly realized by instructions from the CPU 701 shown in FIG. 10, and performs ID authentication by determining whether the image acquisition terminal 2 that sent the authentication request has legitimate authority to receive the service.

[0128] The memory / read unit 99 is mainly realized by instructions from the CPU 701 shown in FIG. 10 and the HDD 705, and stores various data (or information) in the memory unit 9000 and reads various data (or information) from the memory unit 9000.

[0129] [Processing or Operation of the Present Embodiment] Next, the processing or operation of this embodiment will be described with reference to FIGS.

[0130] <Authentication process> First, the authentication process will be described with reference to Fig. 24 to Fig. 27. Fig. 24 and Fig. 25 are sequence diagrams showing the authentication process. Fig. 26 and Fig. 27 are diagrams showing examples of screens of a distributed data processing terminal.

[0131] 24, a request for an authentication destination selection screen is transmitted from the transmitting / receiving unit 61 of the distributed data processing terminal 6 to the service providing server 8 (step S21). This request includes the terminal ID of the distributed data processing terminal 6. At this time, the transmitting / receiving unit 61 also transmits the IP address of its own terminal. As a result, the transmitting / receiving unit 81 of the service providing server 8 receives the request for the authentication destination selection screen and the IP address of the distributed data processing terminal 6.

[0132] Next, the determination unit 82 of the service providing server 8 determines whether or not the terminal ID received in step S21 is associated with a specific session ID and managed in the session management table (see FIG. 20) (step S22). Next, a case where the terminal ID is not managed will be described.

[0133] The transmitting / receiving unit 81 of the service providing server 8 transmits the data of the authentication destination selection screen to the distributed data processing terminal 6 (step S23). As a result, the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the data of the authentication destination selection screen.

[0134] Next, the display control unit 67 of the distributed data processing terminal 6 causes the display 517 to display an authentication destination selection screen s1 as shown in FIG. 26 (step S24). FIG. 26 shows an example of a screen as the distributed data processing terminal 6. The authentication destination selection screen s1 displays an input field b1 for a terminal ID, an input field b2 for a password, and a login button b3 for making a login request (authentication request). Furthermore, the authentication destination selection screen s1 displays authentication server selection buttons a1, a2, and a3 for selecting authentication servers 9a, 9b, and 9c, respectively. For example, the authentication server selection button a1 is a button for when the user receives the service of object detection processing. The authentication server selection button a2 is a button for when the user receives the service of object counting processing.

[0135] Here, when the user inputs his / her terminal ID in the input field b1, his / her password in the input field b2, presses a desired button among the authentication server selection buttons a1, a2, a3, and presses the login button b3, the reception unit 62 receives each input and selection (step S25). Here, a case will be described in which the object detection processing is executed by the object detection processing program ProgD01 out of the image recognition processing by selecting the authentication server selection button a1.

[0136] The transmitting / receiving unit 61 transmits an authentication request for the ID (terminal ID or user ID in this case) to the service providing server 8 (step S26). This authentication request includes the terminal ID and password accepted in step S25, as well as the selection result of the authentication server 9 and the URL of the distributed data processing terminal 6. This authentication result indicates the authentication server ID for identifying the authentication server 9. As a result, the transmitting / receiving unit 81 of the service providing server 8 receives the ID authentication request.

[0137] Next, the memory / read unit 89 of the service providing server 8 searches the authentication server management table (see FIG. 22) using the authentication server ID received as the selection result in step S26 as a search key, and reads out the URL of the corresponding authentication server (step S27).

[0138] Next, the extraction unit 87 extracts only the user ID (part to be authenticated) from the terminal ID received in step S26 (step S28). Then, the transmission / reception unit 81 transmits an ID authentication request to the authentication server 9 indicated by the URL read in step S27 (step S29). This ID authentication request includes the user ID (part to be authenticated) extracted in step S28, the password received in step S26, and the URL of the distributed data processing terminal 6 received in step S26. As a result, the transmission / reception unit 71 of the authentication server 9 receives the user authentication request.

[0139] Next, the storage / read unit 99 of the authentication server 9 uses the pair of the user ID (part to be authenticated) and password received in step S29 as a search key, and the authentication unit 92 performs authentication using the result of searching the authentication management table (see FIG. 23) for the same pair of the part to be authenticated and the password (step S30). If the same pair is managed, the authentication unit 92 judges that the distributed data processing terminal 6 is a valid terminal for receiving a service from the service providing server 8, and if the same pair is not managed, it judges that the distributed data processing terminal 6 is not a valid terminal for receiving a service from the service providing server 8.

[0140] In step S28, the extracting unit 87 extracts the authentication target part from the terminal ID, but this is not limited to this. For example, the service providing server 8 may not have the extracting unit 87, and in step S29, the transmitting / receiving unit 81 may transmit only the user ID (the authentication target part) of the terminal ID in addition to the password and the URL.

[0141] 25, the authentication unit 92 of the authentication server 9 encrypts the token (transmission right) (step S31). Then, the transmitting / receiving unit 91 transmits an authentication result to the distributed data processing terminal 6 based on the URL of the distributed data processing terminal 6 received in step S29 (step S32). This authentication result indicates whether the distributed data processing terminal 6 is valid or not, and includes the token encrypted in step S31. As a result, the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the authentication result of the user. Hereinafter, a case where the user has valid authority will be described.

[0142] The transmitting / receiving unit 61 of the distributed data processing terminal 6 transmits a session establishment request to the service providing server 8 (step S33). This session establishment request includes the terminal ID and the encrypted token received in step S32. As a result, the transmitting / receiving unit 81 of the service providing server 8 receives the session establishment request.

[0143] Next, in order to confirm that the distributed data processing terminal 6 that transmitted the session establishment request has been determined to be valid in step S30, the transmitting / receiving unit 81 of the service providing server 8 transmits a token authentication request to the authentication server 9 (step S34). This token authentication request includes the encrypted token received in step S33. As a result, the transmitting / receiving unit 91 of the authentication server 9 receives the token authentication request.

[0144] Next, the authentication unit 92 decrypts the encrypted token received in step S34 (step S35). The authentication unit 92 then compares the token before encryption in step S31 with the token after decryption in step S35 to authenticate the token (step S36). The transmission / reception unit 91 of the authentication server 9 then transmits the authentication result of step S36 to the service providing server 8 (step S37). As a result, the transmission / reception unit 81 of the service providing server 8 receives the authentication result. Hereinafter, a case where the token is determined to be valid in step S36 will be described.

[0145] Next, the storage / read unit 89 of the service providing server 8 assigns a new session ID in the session management table (see FIG. 20), and manages the session ID in association with the terminal ID and IP address received in step S26 (step S38). Then, the transmission / reception unit 81 transmits data of the service providing screen to the distributed data processing terminal 6 (step S39). As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the data of the service providing screen.

[0146] Next, the display control unit 67 of the distributed data processing terminal 6 displays a service provision screen s2 as shown in Fig. 27 on the display 517 (step S40). Fig. 27 shows an example of a screen of the distributed data processing terminal 6. Here, as an example of the service provided, a remote control service for realizing remote control of the image acquisition terminal 2 from the distributed data processing terminal 6 will be described. The service provision screen s2 shown in Fig. 27 displays an IP address input field c1 for identifying a remote control target, a "Start remote control" button c2, and a "Display area management" button c3.

[0147] <Detection area settings> Next, the process of setting the detection area will be described with reference to Figures 28 to 32 and Figure 38(a). Figures 28 and 29 are sequence diagrams showing the process of setting the detection area.

[0148] First, as shown in Fig. 28, the reception unit 62 of the distributed data processing terminal 6 receives a detection area setting request from a user by pressing the "detection area setting" button c3 shown in Fig. 27 (step S41). Then, the transmission / reception unit 61 transmits the detection area setting request to the near terminal data processing device 5 (step S42). As a result, the transmission / reception unit 51 of the near terminal data processing device 5 receives the detection area setting request.

[0149] Next, the communication unit 58 of the near-terminal data processing device 5 outputs the detection area setting request received in step S42 to the communication unit 48 of the real-time data processing terminal 3 (step S43). As a result, the communication unit 48 of the real-time data processing terminal 3 inputs the detection area setting request.

[0150] Next, the real-time data processing terminal 3 executes the object detection process in the display area management among the real-time processes (step S44).

[0151] (Object detection processing in display area management) Here, the object detection process in the display area management among the real-time processes will be described with reference to Fig. 38(a). Fig. 38(a) is a flowchart showing the object detection process in the display area management among the real-time processes.

[0152] First, the determination unit 33 determines whether the number of imaging elements in the connected imaging unit 40 is one (step S201). In this case, the determination unit 33 makes the determination based on the number of imaging elements read by the process of step S72 described above. Then, if the number of imaging elements is one (step S201; YES), the image processing unit 34 sets the cycle value for repeating the process to 1 / 60 seconds (step S202). In this case, the image processing unit 34 sets the cycle value read by the process of step S73 described above.

[0153] Next, the connection unit 38 acquires one line of captured image data from the imaging unit 40 (step S203). This captured image data is digital image data, for example, 4K image data (width 3840 pixels × height 2160 pixels). In this case, the connection unit 38 executes processing by the image acquisition program (ProgC01 (one line)) shown in FIG. 15(a).

[0154] On the other hand, in the above step S201, if the number of image pickup elements is not one (step S201; NO), the image processing unit 34 sets the cycle value for repeating the real-time processing to 1 / 30 seconds (step S204). In this case, the image processing unit 34 sets the cycle value read out by the processing of the above step S73. By setting the cycle value to 1 / 30 seconds, the cycle value is set large compared to one input, and it is possible to prevent the image synthesis processing described below from being delayed.

[0155] Next, the connection unit 38 acquires two sets of captured image data for two systems from the imaging unit 40 (step S205). The two sets of captured image data are data for hemispherical images as shown in Figs. 4(a) and 4(b), respectively. In this case, the connection unit 38 executes processing according to the image acquisition program (ProgC02 (two systems)) shown in Fig. 15(a).

[0156] Then, the image processing unit 34 synthesizes the two captured image data to generate an equirectangular projection image EC as shown in Fig. 4(c) (step S206). In this case, the image processing unit 34 executes processing according to the synthesis processing program (ProgS02 (synthesis processing)) shown in Fig. 15(b).

[0157] This completes the object detection processing in the display area management in step S44.

[0158] Next, the communication unit 48 of the real-time data processing terminal 3 outputs the captured image data to the communication unit 58 of the near terminal data processing device 5 (step S45). As a result, the communication unit 58 of the near terminal data processing device 5 inputs the captured image data. Then, the memory / read unit 59 of the near terminal data processing device 5 reads out the predetermined area information (center point CP and angle of view α) from the memory unit 5000 (step S46).

[0159] Next, the transmitting / receiving unit 51 of the near-terminal data processing device 5 transmits the photographed image data received in step S45 and the information indicating the center point CP and the angle of view α read in step S46 to the distributed data processing terminal 6 (step S47). As a result, the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the photographed image data and the information indicating the center point CP and the angle of view α.

[0160] Next, the display control unit 67 of the distributed data processing terminal 6 displays a display area setting screen as shown in FIG. 30 on the display 517 based on the captured image data received in step S47 (step S48). FIG. 30 is a diagram showing the display area setting screen. As shown in FIG. 30, the display area setting screen displays a detection area display screen d1 on the left side and an area setting screen d2 on the right side. Here, the display 517 is an example of a display means, and the display control unit 67 is an example of a display control means that displays a screen including a plurality of areas on the display 517. FIG. 31(a) is a diagram showing the position of the detection area in an actual store, and FIG. 31(b) is a diagram showing the relationship between the position of the detection area in the store and the predetermined area information.

[0161] 30 further includes a plurality of (four here) detection area display fields d11 to d14. The detection area display fields d11 to d14 display four predetermined area images Q11 to Q14, which are indicated by four predetermined areas T1 to T4, out of one omnidirectional image (an example of a captured image) obtained by capturing an image of the inside of a store.

[0162] Moreover, the region setting screen d2 displays an attention region setting field d20 in the center. The left, upper, right, and lower sides of the attention region setting field d20 display movement region setting fields d21 to d24, respectively. The user inputs the numbers "1" to "4" displayed at the bottom right of each of the detection region display fields d11 to d14 in each field d20 to d24. As a result, the reception unit 62 receives the input and sets the display position of the movement region relative to the attention region. Here, the detection region display field d11 is set in the attention region setting field d20. Also, the detection region display fields d12, d13, and d14 are set in the movement region setting fields d21, d22, and d23, respectively. As a result, the display control unit 67 displays the region a2 on the left side of the region a1 in the store displayed in the predetermined region image Q11 in the detection region display field d12, based on the predetermined region image Q11 displayed in the detection region display field d11. The display control unit 67 also uses the predetermined area image Q11 displayed in the detection area display field d11 as a reference to display in the detection area display field d13 an area a3 on the back side of the in-store area a1 displayed in the predetermined area image Q11. Furthermore, the display control unit 67 uses the predetermined area image Q11 displayed in the detection area display field d11 as a reference to display in the detection area display field d14 an area a4 on the right side of the in-store area a1 displayed in the predetermined area image Q11.

[0163] Here, the process of changing the area in the store displayed in the detection area display field will be described with reference to Fig. 32. Fig. 32 is a diagram showing a detection range setting screen. When a user selects, by tapping or the like, a predetermined area image Q11 displayed in the predetermined detection area display field (here, detection area display field d11) shown in Fig. 30, the reception unit 62 receives the selection. As a result, the display control unit 67 displays a display area setting screen as shown in Fig. 32 on the display 515 of the distributed data processing terminal 6.

[0164] 32, the detection range setting screen displays a detectable range P, a set button d8, and a cancel button d9. In the detectable range P, the predetermined area image Q11 of the selected detection area display field d11 is actually displayed.

[0165] The detectable range P is a range in which the shooting range of the imaging unit 40 can detect an object (such as a face). This detectable range P includes a matchable range C in which the image of the object can be matched, and an unmatchable range R outside the matchable range C in which matching is not possible. In Fig. 32, margins a and b are predetermined values ​​provided at the left and right ends in the x direction and the top and bottom ends in the y direction in the detectable range P, and are determined in advance.

[0166] The reason why the matchable range C is provided separately from the detectable range P is that since the detectable range P is fixed as the imaging range of the imaging element 401a etc., the matchable range C can be changed by changing the margin a and margin b, and an object can be detected in a range that matches the size of any room or area. Also, this eliminates the need for the real-time data processing terminal 3a to transmit the matched data to the centralized data processing server 7 via the near-terminal data processing device 5 and the distributed data processing terminal 6 when an object goes out of the detectable range P before entering the matchable range C after entering the detectable range P, thereby reducing unnecessary communication charges.

[0167] The "Set" button d8 is a button that the user presses when deciding on the detection area. The "Cancel" button d9 is a button that the user presses when deciding on the detection area is to be cancelled.

[0168] Furthermore, the detectable range P indicates the same range as the predetermined area T (see FIG. 7). Therefore, setting the detectable range P is equivalent to setting the predetermined area T. If the aspect ratios of the detectable range P and the predetermined area T are different, the display control unit 67 displays the predetermined area T in accordance with the size of the detectable range P by enlarging or reducing it while maintaining the aspect ratio of the predetermined area T. For example, if the aspect ratio of the detectable range P is 9:16 and the aspect ratio of the predetermined area T is 3:4, the display control unit 67 displays the predetermined area T so that the width of the predetermined area T is maximized within the detectable range P. This detectable range P can be changed by the user at the distributed data processing terminal 6. This change and setting will be described with reference to FIG. 29.

[0169] First, the reception unit 62 of the distributed data processing terminal 6 receives an input from a user via the touch panel 521 or the like (S51). The types of inputs are changing the image of the predetermined area within the detectable range P by swiping, pinching in or out, pressing the "Set" button d8, or pressing the "Cancel" button d9 on the detection range setting screen shown in Fig. 32. Changes to the detectable range P include, in Fig. 7(a), displaying a different predetermined area within the three-dimensional sphere CS by swiping (movement), displaying a wide range of predetermined areas including the predetermined area T by pinching in (reduction), and displaying a narrow range of predetermined areas within the predetermined area T by pinching out (enlargement).

[0170] Here, an example of changing the predetermined area image displayed in the detectable range P will be described with reference to Fig. 33. Fig. 33(a) is a diagram showing the predetermined area image Q11 before change in the detectable range P, Fig. 33(b) is a diagram showing the predetermined area image Q111 after change in the detectable range P, and Fig. 33(c) is a diagram showing an example of a warning display for the predetermined area image Q112 after change in the detectable range P.

[0171] As shown in FIG. 33(a), when the user swipes the hand h1 to the right, the reception unit 62 receives a change to the detectable range P. As a result, the display control unit 67 changes the predetermined area image Q11 displayed in the detection area display field d11 to a predetermined area image Q111. Then, when the user presses the "Set" button d8, the reception unit 62 receives the setting, and the display control unit 67 displays the display area setting screen shown in FIG. 30. In this case, the detection area display field d11 displays the predetermined area screen Q111 shown in FIG. 33(b).

[0172] Furthermore, if the user swipes the hand h1 to the right in the state of Fig. 33(b), causing the display control unit 67 to display an image (predetermined area image Q112) that is the same as the predetermined area image Q12 on the detectable range P, the positional relationship between each of the predetermined area images Q11 to Q14 on the detection area display screen d1 shown in Fig. 30 and each of the area numbers "1" to "4" on the area setting screen d2 will no longer match. Therefore, as shown in Fig. 33(c), the display control unit 67 displays a warning by superimposing a masking image (for example, an image with a semi-transparent red background color) on the predetermined area image Q112.

[0173] Here, the process performed by the display control unit 67 when displaying a warning will be described in more detail. FIG. 31(b) is a diagram showing the relationship between the position of the detection area in the store and the predetermined area information. If the virtual camera IC shown in FIG. 8 is installed at the center of the circle C and the elevation angle "ea" in FIG. 8 is fixed, the center point CP of each area can be determined by the azimuth angle "aa" as shown in FIG. 31(b). In FIG. 31(b), the center points for the areas a1 and a2 in the store are indicated by ● for convenience.

[0174] In this embodiment, as an example, a method will be described in which the display control unit 67 that displays a warning compares the azimuth angles of the attention area a1 and the movement areas a2, a3, and a4 to determine whether the arrangement condition set by the user on the area setting screen of FIG. 30 is satisfied. For example, the azimuth angle "aa1" of the area a1 in the store shown in FIG. 31(b) is set as the starting point (0 degrees), and the angle increases counterclockwise. If the arrangement relationship of the areas a1, a2, a3, and a4 in the store set by the user is as shown in FIG. 31(b) and the azimuth angles of the areas a2, a3, and a4 in the store are aa2, aa3, and aa4, respectively, the condition that satisfies this arrangement is aa1 < aa2 < aa3 < aa4. As in the case of FIG. 31(b), if the above-mentioned condition is not satisfied when the predetermined image of any of the movement areas a2, a3, and a4 is changed, the display control unit 67 displays a warning as shown in FIG. 33(c).

[0175] The judgment by the display control unit 67 when displaying a warning is not limited to the above example. For example, instead of setting the condition for satisfying the arrangement as aa1 < aa2 < aa3 < aa4, a predetermined threshold value for the horizontal angle aa may be set for each of the regions a1, a2, a3, and a4, and whether or not to display a warning may be judged depending on whether or not each threshold value is exceeded. Furthermore, the above-mentioned threshold value may be set as a common value for each of the regions a1, a2, a3, and a4, and the judgment method (judgment condition) may be changed to process whether or not to display a warning.

[0176] 29, if the type of input is a change in the detectable range P (S52; change), the display control unit 67 changes and displays the detectable range P according to the amount of pinch-in or pinch-out operation (S53). Then, the process returns to step S51.

[0177] Furthermore, if the type of input is confirmation of the detection area by pressing the "Set" button d8 (S52; Confirm), the transmitting / receiving unit 61 transmits a confirmation request to the near terminal data processing device 5 (S54). This confirmation request includes predetermined area information indicating the center point CP and the angle of view α for identifying the detectable range P (predetermined area T) confirmed in steps S51 and S52. Note that, if the detectable range P (predetermined area T) is confirmed after being changed in steps S51 and S52, the predetermined area information indicating the center point CP and the angle of view α after the change is included. As a result, the transmitting / receiving unit 51 of the near terminal data processing device 5 receives the confirmation request.

[0178] Next, in the near terminal data processing device 5, the storage / read unit 59 stores the determined predetermined area information in the storage unit 5000 (S55).

[0179] Also, when the input type is the pressing of the "Cancel" button d9 to cancel the setting of the detection area (S52; Cancel), the transmitting / receiving unit 61 transmits a shooting stop request to stop shooting to the near terminal data processing device 5 (S56). As a result, the transmitting / receiving unit 51 of the near terminal data processing device 5 receives the shooting stop request. Then, the communication unit 58 of the near terminal data processing device 5 outputs the shooting stop request to the communication unit 48 of the real-time data processing terminal 3 (S57). As a result, the communication unit 48 of the real-time data processing terminal 3 inputs the shooting stop request. Then, in the real-time data processing terminal 3, the connection unit 38 causes the imaging unit 40 to stop shooting (S58). After the process of step S56, the display control unit 67 returns to the display of the detection range setting screen of FIG. 30 (S59).

[0180] This completes the process of setting the detection area.

[0181] <Preparation for image recognition> Next, a preparation process for image recognition will be described with reference to Fig. 34 to Fig. 36. Fig. 34 is a sequence diagram showing the process of a request to start image recognition.

[0182] As shown in FIG. 34, in the distributed data processing terminal 6, the reception unit 62 receives a start request for image recognition from a user (step S61). In this case, a GUI (Graphical User Interface) of the distributed data processing terminal 6 is used. As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits start request information for image recognition indicating a start request for image recognition to the near-terminal data processing device 5 (step S62). This start request information includes an authentication server ID for identifying the authentication server 9 that performed the authentication in the above-mentioned authentication process (see FIG. 24 and FIG. 25). As a result, the transmission / reception unit 51 of the near-terminal data processing device 5 receives the start request information for image recognition. Then, the communication unit 58 transmits the start request information for image recognition to the real-time data processing terminal 3 (step S63). As a result, the communication unit 48 of the real-time data processing terminal 3 receives the start request information for image recognition. In this way, by separating the user interface from the real-time data processing terminal 3, remote operation from the distributed data processing terminal 6 is possible.

[0183] FIG. 35 is a sequence diagram showing preparation processing for real-time processing by the real-time data processing terminal. 35, the connection unit 38 of the real-time data processing terminal 3 acquires the model number of the imaging unit 40 from the imaging unit 40 (step S71). In this case, the connection unit 38 requests the imaging unit 40 for the model number, and the imaging unit 40 transmits its own model number in response to this request.

[0184] Next, the storage / reading unit 39 searches the imaging element information management DB 3001 (see FIG. 14(a)) using the model number acquired in step S71 as a search key to read out the corresponding number of imaging elements and the type of lenses (step S72). Furthermore, the storage / reading unit 39 searches the cycle value management DB 3002 (see FIG. 14(b)) using the number of imaging elements read in step S72 as a search key to read out the corresponding cycle value (step S73).

[0185] Next, the storage / readout unit 39 searches for an image acquisition program managed in the image acquisition program management DB 3003 (see FIG. 15(a)) using the number of imaging elements read out in step S72 as a search key (step S74). Next, the storage / readout unit 39 searches for a synthesis processing program managed in the synthesis processing program management DB 3004 (see FIG. 15(b)) using the number of imaging elements read out in step S72 as a search key (step S75). Next, the storage / readout unit 39 searches for a distortion correction program managed in the distortion correction program management DB 3005 (see FIG. 15(c)) using the type of lens read out in step S72 as a search key (step S76). Next, the storage / readout unit 39 searches for a service program managed in the service program management DB 3006 (see FIG. 15(d)) using the type of lens read out in step S72 as a search key (step S77).

[0186] Next, based on the search results of steps S74 to S77, the determination unit 33 determines whether all of the programs to be executed have been installed, depending on whether the programs to be executed are managed (step S78). For example, in step S74, the storage / readout unit 39 searches for the image processing programs, and if the image processing programs are managed, the determination unit 33 determines that the programs are installed, whereas if the image processing programs are not managed, the determination unit 33 determines that the programs are not installed.

[0187] Then, when the judgment unit 33 judges that all four programs are managed (step S78; YES), the process shown in Fig. 35 ends. On the other hand, when the judgment unit 33 judges that at least one of the four programs is not installed (step S78; NO), the process proceeds to step S91 described later.

[0188] Fig. 36 is a sequence diagram showing a program acquisition process, in which the real-time data processing terminal 3 acquires, from the service providing server 8, a program that has been found not to be installed by the process shown in Fig. 35.

[0189] 36, the communication unit 48 of the real-time data processing terminal 3 transmits request information indicating a request for a program that is not installed to the communication unit 58 of the near-terminal data processing device 5 (step S91). This request information includes the name of the program to be requested.

[0190] Next, the transmitting / receiving unit 51 of the near-terminal data processing device 5 transmits the request information received by the communication unit 58 to the transmitting / receiving unit 61 of the distributed data processing terminal 6 (step S92). Then, the transmitting / receiving unit 61 of the distributed data processing terminal 6 transmits the request information to the transmitting / receiving unit 81 of the service providing server 8 (step S93).

[0191] Next, in the service providing server 8, the storage / reading unit 89 reads out the program indicated by the program name included in the request information (step S94). Then, the transmitting / receiving unit 81 transmits the read out program to the transmitting / receiving unit 61 of the distributed data processing terminal 6 (step S95). At this time, the program name is also transmitted.

[0192] Next, the transmitting / receiving unit 61 of the distributed data processing terminal 6 transmits the program including the program name to the transmitting / receiving unit 51 of the near terminal data processing device 5 (step S96). Then, the communication unit 58 of the near terminal data processing device 5 transmits the program including the program name to the communication unit 48 of the real-time data processing terminal 3 (step S97).

[0193] Next, the storage / reading unit 39 of the real-time data processing terminal 3 installs the program acquired by the communication unit 48, and manages the name of the program to be installed in each table shown in FIG. 15 (step S98).

[0194] Next, the storage / readout unit 39 starts all programs necessary for the image recognition processing (step S99). As a result, the real-time data processing terminal 3 executes the started programs, thereby starting the real-time processing shown below.

[0195] <Image recognition processing> Here, a case will be described in which the "Login to object detection processing service" button b1 shown in Fig. 26 is pressed and the service providing server 8 executes the object detection processing service among the image recognition processing services. Fig. 37 is a sequence diagram showing the image recognition processing. First, the real-time data processing terminal 3 performs real-time processing (step S111).

[0196] (Object detection processing) Here, the object detection process among the real-time processes will be described with reference to Fig. 38(a) and (b). Note that the process of Fig. 38(b) is performed after the process of Fig. 38(a), but since the process of Fig. 38(a) has already been described, the process of Fig. 38(b) will be described.

[0197] After the process of step S203, the object detection unit 35 detects object candidates in the captured image data, thereby detecting the object (step S207). In this case, the image processing unit 34 executes the process by a service program (ProgD01 (object detection process)) shown in FIG. 15(d). The image processing unit 34 searches for a rectangle from the edge of the captured image while comparing it with shape model data of the object stored in advance in the storage unit 3000, and selects the position of the most likely object feature point. For the process of step S207, for example, a known technology such as that disclosed in "Face Recognition Technology and Its Applications: Special Issue on Elemental Technologies and Solutions Supporting Public Safety; Biometrics Authentication, NEC Technical Journal, Vol. 63, no. 3, pp. 26-30, 2010-09" can be used.

[0198] Next, the image processing unit 34 corrects distortion of the image of the detected object (step S208). In this case, the image processing unit 34 executes the process according to the image processing program (ProgW01 (wide-angle distortion correction)) shown in FIG. 15(c).

[0199] After the process of step S206, the process proceeds to step S207, where the object detection unit 35 detects feature points that are candidates for the object in the data of the equirectangular projection image EC, thereby detecting the object. Next, in step S208, the image processing unit 34 corrects the distortion of the image of the detected object. In this case, the image processing unit 34 executes the process using the image processing program (ProgW02 (fisheye distortion correction)) shown in FIG. 15(c).

[0200] As a result, as shown in FIG. 37, the communication unit 48 of the real-time data processing terminal 3 transmits the captured image data to the communication unit 58 of the near-terminal data processing device 5 (step S112). Then, the transmission / reception unit 51 of the near-terminal data processing device 5 transmits the captured image data received in step S112 to the transmission / reception unit 61 of the distributed data processing terminal 6 (step S113). As a result, the display control unit 67 of the distributed data processing terminal 6 displays the captured image shown in FIG. 45 on the display 517 in real time (step S114). FIG. 45 is a diagram showing an example of a display of a captured image in the distributed data processing terminal. In this captured image, a frame indicating a rectangle where an object (here, a face) has been detected is displayed. Steps S112 to S114 are streaming processing.

[0201] (Event generation process) Next, the event generation process of the real-time processing will be described with reference to Figures 39 and 40. Figures 39 and 40 are flow charts showing the event generation process of the real-time processing.

[0202] If the object detection unit 35 does not detect an object (here, a face) by the process of step S207 (S211; NO), the process returns to the process of step S211 again after waiting for a predetermined time (a set cycle value) (S212). For example, detection is performed 30 times per second.

[0203] On the other hand, if the object detection unit 35 detects an object (here, a face) by the process of step S207 described above (S211; YES), the process proceeds to step S213. Also, if the position of the detected object is not within the collation possible range C (outside the collation possible range C) (S211; NO), the process proceeds to step S212.

[0204] In step S213, the storage / reading unit 39 stores the object information on the object detected in step S207 as a new record in the object information management table (see FIG. 16). In this case, the storage / reading unit 39 erases the end flag for the record already stored in the object information management table in order to reset it.

[0205] Next, the memory / read unit 39 erases the end flags for all records in the object displacement management table in order to initialize the end flags stored in the object displacement management table (S214). Note that the end flags are flags that are added in step S216, which will be described later.

[0206] Next, if there are records remaining to be read in the object displacement management table (S215; YES), the storage / reading unit 39 reads data of one record from the object displacement management table (S216). On the other hand, if all records have been read (S215; NO), the process proceeds to step S220, which will be described later.

[0207] Next, the calculation unit 31 calculates the distance between the latest (current) object detection position stored in step S213 and the past object detection position (here, coordinates (x, y)) stored in step S216 (S217). Then, the judgment unit 33 judges whether the distance calculated in step S217 is shorter (less than) a predetermined threshold (S218). In this embodiment, the calculation is performed using the square Euclidean distance shown in the following (Equation 2). Note that the judgment unit 33 may also judge whether it is equal to or less than the predetermined threshold.

[0208]

number

[0209] The squared Euclidean distance has the advantage that it is easy to find the position of a past object that is close because the more distant the object is, the greater the weight. Note that other well-known distance calculation methods can also be used, such as city block distance or counting the number of pixels shifted in only a specific direction (x direction or y direction).

[0210] Next, if the distance calculated in step S217 is shorter than a predetermined threshold (S218; YES), the distance between the current object position and the past positions of the remaining records managed in the object displacement management table is not calculated, and the process proceeds to step S219. If the distance calculated in step S217 is not shorter (the same as or longer) than the predetermined threshold (S218; NO), the process returns to the process of step S215.

[0211] Next, the memory / read unit 39 updates the object displacement management table by replacing the position information indicating the past object position whose distance was determined to be shorter than the threshold with the position information indicating the latest object position whose distance was determined to be shorter than the threshold (S219). In this case, the memory / read unit 39 adds an "end flag" to the end flag column of the record including the position information used for replacement in the object information management table and the object displacement management table.

[0212] Next, if there are any records remaining in the object displacement management table that do not have an end flag set (S220; YES), the communication unit 48 of the real-time data processing terminal 3 outputs a display change request to the communication unit 58 of the near-terminal data processing device 5, indicating a request for a display change (S211).

[0213] Here, the process of displaying the movement count number and storing the movement information will be described with reference to Fig. 41. Fig. 41 is a sequence diagram showing the process of displaying the entry / exit count number and storing the movement information.

[0214] First, the communication unit 48 of the real-time data processing terminal 3 transmits a display change request indicating a display change request to the communication unit 58 of the near terminal data processing device 5 in the above step S221. This display change request includes an area number indicating an area in the store where an object as a person was present, and an object ID. As a result, the communication unit 58 of the near terminal data processing device 5 inputs the display change request. Then, the transmission / reception unit 51 transmits (transfers) the display change request to the distributed data processing terminal 6 (S302). As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the display change request.

[0215] Next, the distributed data processing terminal 6 performs a change display process (S303). Here, the process of step S303 will be described in detail with reference to FIG.

[0216] The storage / read unit 69 of the distributed data processing terminal 6 searches the object presence management DB 6001 (see the object presence management table in FIG. 18(a)) using the object ID received in step S302 as a search key (S303-1). Then, the judgment unit 63 judges whether or not an object ID identical to the object ID of the search key is stored in the object presence management DB 6001 (S303-2).

[0217] Then, if the object ID is not stored in the object presence management table (S303-2; NO), the storage / reading unit 69 further reads out the number of the “area of ​​interest” from the display area management DB 6002 (see the display area management table in FIG. 18(b)) (S303-3).

[0218] Next, the judgment unit 63 judges whether the area number received in step S302 matches the number of the attention area read out in step S303-3 (S303-4). If they do not match (S303-4; NO), the process of FIG. 42 ends. On the other hand, if they match (S303-4; YES), a new record is added to the object presence management table of the object presence management DB 6001, and the received person number is stored (S303-5). In this embodiment, the area number "1" of the entrance of the store or the like is set as the attention area, but a different area number can be set as the attention area by storing a display area management table according to the layout of the store or the area to be counted. After the process of step S303-5, the process of FIG. 42 ends.

[0219] Next, if the object ID is stored in the object presence management table in step S303-2 (S303-2; YES), the storage / reading unit 69 reads out the number of the area of ​​interest from the display area management DB 6002 (S303-6).

[0220] Next, the judgment unit 63 judges whether the area number received in step S302 is different from the number of the attention area read out in step S303-6 (S303-7). If the area number is not different (the same) as the number of the attention area (S303-7; NO), the process of FIG. 42 ends. Note that the movement of a person when the area number and the number of the attention area match is the movement when the person once leaves the area a1 in the store indicated by the attention area "1", but returns to the area a1 again without moving to another area a2 in the store indicated in FIG. 31. Note that in this embodiment, the above-mentioned attention area "1" is treated as the first area identification information for identifying the first area, and the other attention area numbers are treated as the second area identification information for identifying the second area.

[0221] On the other hand, if the area number is different from the number of the attention area (S303-7; YES), the storage / reading unit 69 searches the display area management DB 6002 using the area number received in step S302 as a search key to read out the corresponding moving area information (left, top, right, bottom) (S303-8). Then, if the moving area is "left" (S303-9; left), the calculation unit 65 counts up to indicate a move to the left (S303-10). Also, if the moving area is "up" (S303-9; up), the calculation unit 65 counts up to indicate a move to the top (S303-11). Furthermore, if the moving area is "right" (S303-9; right), the calculation unit 65 counts up to indicate a move to the right (S303-12). In this example, the area below the attention area is outside the store, so it is not counted up.

[0222] Next, the display control unit 67 changes the display of the count display area m3 in the upper right portion of the matching result display screen (S303-13), as shown in Fig. 45. In this example, area number 1 is set as the reference (start area), and it is displayed that up to now, one person has moved to the left area a2, two people have moved to the upper (back) area a3, and three people have moved to the right. Note that the "display movement information" button e1 displayed in the lower right of the matching result display screen is a button for displaying the number of counts per unit time, which will be described later.

[0223] This ends the process of step S303 (see FIG. 42).

[0224] 41, the transmitting / receiving unit 61 of the distributed data processing terminal 6 transmits an object position storage request indicating a request to store the position of an object such as a person to the centralized data processing server 7 (S304). This object position storage request includes information on the date and time when the distributed data processing terminal transmits the object position storage request, in addition to the area number and object ID received in step S302. As a result, the transmitting / receiving unit 71 of the centralized data processing server 7 receives the object position storage request.

[0225] Next, in the centralized data processing server 7, the memory / read unit 79 associates the information indicating the date and time, area number, and object ID received in step S304 with each other and stores them as one record in the object position management DB 7002 (see Figure 19 (b)) (S305).

[0226] It should be noted that one record in which the date and time, the area number, and the object ID are associated may be deleted after a certain period of time (for example, three months) has passed since it was stored.

[0227] 39, after the process of step S221, the storage / readout unit 39 deletes the record to which the end flag is not set (S222), and proceeds to the process of step S231 shown in Fig. 40. This process is for deleting the record from the object displacement management table to exclude it from future comparison targets, since if the record remains in the object displacement management table without the end flag set, it can be assumed that the object (here, a person) has already moved out of the collation possible range C described below.

[0228] On the other hand, if there are no records left in the object displacement management table to which no end flag is attached (S220; NO), the process of step S221 is omitted and the process proceeds to step S231.

[0229] Next, if there are records in the object information management table to which no end flag has been added (S231; YES), the storage / reading unit 39 adds the position information of these remaining records to the object displacement management table as new records (S232). In this case, the storage / reading unit 39 manages the matching progress status of the added record as "0 (before recognition)". This process is for managing the subsequent displacement, since it can be assumed that a new object (here, a person) has entered the detectable range P described below.

[0230] In this case, the storage / reading unit 39 adds "+1" to the value of the previously added ID and stores it in the ID field of the added record. Note that the storage / reading unit 39 does not renumber the IDs when deleting a record in step S221. In this way, it is possible to save the correct result for a record whose matching progress status is "1 (matching in progress)".

[0231] On the other hand, if there are no records to which an end flag is not added in the object information management table (S231; NO), the process proceeds to step S236, which will be described later.

[0232] Next, after the process of step S232, the judgment unit 33 judges whether or not an object with a matching progress status of "0 (before matching)" in the object displacement management table is within the matching possible range C (S233). This judgment is made based on the position information of the object. Then, if it is judged in step S233 that the object is within the matching possible range C (S233; YES), the image processing unit 34 refers to the position, width, and height of the object within the matching possible range C in the object displacement management table, cuts out partial image data determined by these values, and encodes it in a general-purpose format such as JPEG (S234). In this case, the storage / reading unit 39 changes the matching progress status of the corresponding record in the object displacement management table to "1 (matching in progress)".

[0233] On the other hand, in step S233, if it is determined that the match is not within collation range C (S233; NO), the process proceeds to step S236.

[0234] Next, the event generating unit 36 ​​generates an event message notifying the near-terminal data processing device 5 of the transmission of the partial image data and the image ID of the partial image associated with the partial image data (S235). Specifically, the event generating unit 36 ​​generates an event message such as "Send".

[0235] Next, the image processing unit 34 clears the displayed text image, and then refers to the position in the object displacement management table associated with the matching progress status of "1 (matching in progress)" or "2 (matching completed)," and synthesizes the text image at the corresponding position in the captured image (S236).

[0236] FIG. 45 shows an example of combining characters with an image. FIG. 45 shows the range of photography by the imaging unit when an image acquisition terminal is installed in a room (here, the entrance of a building). The image processing unit 34 combines a standby state mark m1 represented by "!" with the partial image at the position (x2, y2) to indicate that matching is in progress. The image processing unit 34 also combines object ID information m2 represented by "Kato" with the corresponding position of the partial image by referring to the name to indicate the matching result with respect to the partial image at the position (x3, y3). This object ID information m2 is information read out from the object displacement management DB 3008. The combined image data is transmitted to the distributed data processing terminal 6 by steps S112 and S113 shown in FIG. 37, and is displayed in step S114.

[0237] Next, if the communication unit 48 has not received the collation result in step S318 described below (S237; NO), the process proceeds to step S212.

[0238] On the other hand, when the communication unit 48 receives the matching result (object ID and image ID of the partial image) (S237; YES), the storage / readout unit 39 stores the received name in the name field associated with the image ID of the received partial image in the object displacement management table (S238). In this case, the storage / readout unit 39 changes the matching progress status from "1 (matching in progress)" to "2 (matching completed)". Then, the image processing unit 34 clears the displayed character image, and then refers to the position in the object displacement management table associated with the image ID of the received partial image, and synthesizes the character image with the received name at the corresponding position of the captured image (S239). Then, the process returns to step S212. In step S239, the display control unit 37 updates the display of "!" that was being matched to a display of the name, such as "Kato". This allows the user of the distributed data processing terminal 6 to quickly know the progress status of the matching.

[0239] This completes the event generation process in the real-time process.

[0240] Next, returning to FIG. 37, the communication unit 48 of the real-time data processing terminal 3a transmits an event message, partial image data of the object (here, a face), and the image ID of the partial image to the communication unit 58 of the near-terminal data processing device 5 (S115).

[0241] Next, in the near terminal data processing device 5, the data detection unit 56 detects whether or not a "Send" event message has been received by the communication unit 58 (S116). Then, if reception of an event message is detected (S116; YES), the communication unit 58 receives the partial image data and the image ID of the partial image transmitted together with the event message (S117). Then, the storage / readout unit 59 temporarily stores the partial image data and the image ID of the partial image in the storage unit 5000 (S118).

[0242] Next, the data detection unit 56 monitors whether or not the reception of the partial image data is completed (S119). The process of this step S119 is repeated until all partial image data and image IDs of partial images are received for one event message (S119; NO). Then, when the reception of the partial image data is completed (S119; YES), the storage / readout unit 59 reads out all partial image data and image IDs of partial images sent together with one event message from the storage unit 5000 and temporarily stored in the storage unit 5000 (step S120). Thereafter, the transmission / reception unit 51 transmits all partial image data and image IDs of partial images read out in step S120 to the transmission / reception unit 61 of the distributed data processing terminal 6 via the intranet 200 (S121). As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 receives all partial image data and image IDs of partial images. The partial image data is used as data to be matched later.

[0243] (Matching process) Next, the matching process will be described with reference to Fig. 43 to Fig. 45. Fig. 43 is a sequence diagram showing the matching process. Fig. 44 is a flowchart showing the matching process. Fig. 45 is a diagram showing the shooting range of an imaging unit when an image acquisition terminal is installed in a room.

[0244] First, when the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the partial image data as the data to be matched and an object ID (object ID related to the partial image) for identifying the object shown in the partial image by the process of step S121 described above, the transmitting / receiving unit 61 transmits matching request information indicating a matching request to the centralized data processing server 7 (S313). This matching request information includes the data to be matched and the object ID related to the partial image. As a result, the transmitting / receiving unit 71 of the centralized data processing server 7 receives the matching request information.

[0245] Next, the centralized data processing server 7 performs a matching process (S314). Here, the matching process will be described with reference to FIG.

[0246] As shown in FIG. 44, the feature generating unit 74 of the centralized data processing server 7 performs conversion to decode the matched data received in step S313 into bitmap data, and generates feature parameters for identifying an individual, such as the contours and inclinations of the eyes and nose of the face image related to the matched data (S401).

[0247] Next, the storage / readout unit 79 searches the matching data management DB 7001 to determine whether or not the registered matching data remains (S402). If it is determined that the registered matching data remains, the matching unit 75 compares the parameters of the feature amounts of both data (matching data and matched data) to calculate the similarity (S403). Then, the storage / readout unit 79 temporarily stores the object ID attached to the matching data and the "similarity" calculated in step S403 as a pair in the storage unit 7000 (S404). After that, the matching data is changed to the next row in the matching data management table shown in FIG. 19(a), and the process of step S402 is performed again.

[0248] On the other hand, if it is determined in step S402 that no matching data remains (including the case where there is no matching data at all), the process proceeds to step S405. Then, the determination unit 73 determines whether the maximum similarity among the similarities temporarily stored in the storage unit 7000 is greater than a threshold value (S405).

[0249] Then, in step S405, if the judgment unit 73 judges that the maximum similarity is greater than the threshold (YES), the storage / reading unit 79 reads out the object ID attached to the matching data with the maximum similarity from the matching data management DB 7001 (S406). The threshold is, for example, "80%." As a result, when the similarity is low, the object ID is not read out.

[0250] On the other hand, in step S405, if the judgment unit 73 judges that the maximum similarity among the similarities temporarily stored in the storage unit 7000 is equal to or less than the threshold value (NO), the storage and reading unit 79 registers the data to be matched as matching data (S407). If matching data has already been registered, the storage and reading unit 79 sets "the number of current matching data + 1" as an object ID and registers the matching data in association with this object ID. Next, the feature generation unit 74 generates features of the registered matching data (S408). Then, the storage and reading unit 79 reads out the object ID associated in step S407 (S409).

[0251] With this, the processing in FIG. 44 ends.

[0252] 43, the transmitting / receiving unit 71 of the centralized data processing server 7 transmits matching result information indicating the matching result to the distributed data processing terminal 6 (S315). This matching result information includes the object ID read in step S406 or step S409 or the information read in step S407, and the object ID related to the partial image received in step S313. As a result, the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the matching result information.

[0253] Next, in the distributed data processing terminal 6, the display control unit 67 displays the matching result as shown in Fig. 45 (S316). Here, the display of the matching result will be specifically described with reference to Fig. 45. Fig. 45 is a diagram showing a matching result display screen.

[0254] As with the detection range setting screen shown in Fig. 32, the detectable range P on the matching result display screen is the range within which the shooting range of the imaging unit 40 can detect an object (here, a face). This detectable range P includes a matchable range C in which the image of the object (here, a face) can be matched, and an unmatchable range R outside this matchable range C in which matching is not possible. In Fig. 45, margins a and b are predetermined values ​​provided at the left and right ends in the x direction and the top and bottom ends in the y direction, respectively, in the detectable range P. These values ​​are determined in advance.

[0255] The reason why the matchable range C is provided separately from the detectable range P is that the detectable range P is fixed as the imaging range of the imaging element 401a, etc., and therefore it is possible to detect an object within a range that matches the size of any room or area. In addition, this eliminates the need for the real-time data processing terminal 3a to transmit the matched data to the centralized data processing server 7 via the near-terminal data processing device 5 and the distributed data processing terminal 6 when an object enters the detectable range P and then leaves the detectable range P before entering the matchable range C, thereby reducing unnecessary communication charges.

[0256] The face detection by the process of step S207 described above is performed throughout the entire detectable range P, which is the shooting range. If the object is outside the rectangular range determined by the margins a and b, the object is not within the matchable range C. In this case, the object is detected within the detectable range P but not within the matchable range C. As a result, the object is present in the unmatchable range R, which is the range within the detectable range P other than the matchable range C.

[0257] Here, (x1, y1) indicates the coordinates of the upper left corner of the partial image data when an object existing within the collation impossible range R is detected.

[0258] (x2, y2) indicates the coordinates of the upper left corner of the partial image data when an object present within the collation possible range C is detected. Also, here, a standby state mark m1 represented by "!" is displayed. This standby state mark m1 is displayed from the distributed data processing terminal 6 transmitting the collation request information (S313) until it receives the collation result information from the centralized data processing server 7 (S315).

[0259] (x3, y3) indicate the coordinates of the upper left corner of the partial image data when an object present within the collation possible range C is detected. Also, here, object ID information m2 represented by "S001" is displayed. This object ID information is the "partial image object ID" received from the centralized data processing server 7 in step S315. In this way, when an object enters the collation possible range C, a standby state mark m1 is displayed at first, but after 1 to 2 seconds, object ID information m2 is displayed. This allows the viewer to recognize the displayed object (here, a face).

[0260] Furthermore, as described above, the count display area m3 is displayed in the upper right corner of the matching result display screen. This makes it easy to see that, in Fig. 31, one person who entered the store from the entrance moved from area a1 to area a2, two people moved from area a1 to area a3, and three people moved from area a1 to area a4.

[0261] 43, the transmitting / receiving unit 61 of the distributed data processing terminal 6 transmits (transfers) the collation result to the near terminal data processing device 5 (S317). This collation result includes the same information as in step S315. As a result, the transmitting / receiving unit 51 of the near terminal data processing device 5 receives the collation result.

[0262] Next, the communication unit 58 of the near-terminal data processing device 5 transmits (transfers) the collation result to the communication unit 48 of the real-time data processing terminal 3 (S318). This collation result includes the same information as in step S317. As a result, the communication unit 48 of the real-time data processing terminal 3 receives the collation result.

[0263] (Display travel information) Next, the process of displaying movement information will be described with reference to Fig. 46 to Fig. 48. Fig. 46 is a sequence diagram showing the process of displaying movement information. Fig. 47 is a flowchart showing the process of counting movements to a predetermined area in a building (for example, areas a1 to a4 in Fig. 31) in consideration of the moving direction of an object. Fig. 48 is a diagram showing a movement result display screen.

[0264] First, when a user presses the "Display movement information" button e1, the reception unit 62 of the distributed data processing terminal 6 receives the display of the movement information (S501). Then, the transmission / reception unit 61 transmits a movement information request to the centralized data processing server 7 to request the movement information (S502). This movement information includes the current date and time and a display area management table. As a result, the transmission / reception unit 71 of the centralized data processing server 7 receives the movement information request.

[0265] Next, the counting unit 76 of the centralized data processing server 7 counts the number of records by a combination of an area number indicating an area in the store to which the user has moved per unit time (per hour in this embodiment) and an object ID (S504). Here, the process of step S504 will be described in detail with reference to FIG.

[0266] First, the storage / readout unit 79 refers to the area number column of the object position management table of the object position management DB 7002, and starts searching from the top row downwards for a record in which an area number matching the number in the attention area column of the display area management DB 601 received in step S502 is managed (S504-1). Then, if the date and time in the date and time column of the searched record is not within the range to be read (or is outside the range) (S504-2; NO), the processing of FIG. 47 ends. For example, if the received current date and time is April 18, 2019, the same "day" is within the range and a different "day" is outside the range. On the other hand, if the date and time in the searched record (an example of a first record) is within the range to be read (S504-2; YES), the storage / readout unit 79 reads out the object ID of the searched record, and searches for a record (an example of a second record) in which a matching object ID is managed in the subsequent records (S504-3). If there is a record in which a matching object ID is managed and the date and time managed in this record is within the read range (S504-4; YES), proceed to step S504-5. On the other hand, if there is no record in which a matching object ID is managed, or there is a record in which a matching object ID is managed but the date and time managed in this record is not within the read range (S504-4; NO), return to step S504-1.

[0267] Next, if there is a record in which a matching object ID is managed and the date and time managed in this record is within the read range (S504-4; YES), the storage / read unit 79 searches the time zone column of the counting management DB 7003 based on the date and time managed in the record searched for in step S504-3 above, and identifies the record to be counted (S504-5). In this embodiment, there are time zone records every hour, and for example, if the time zone of the searched record is between 8:00 a.m. and 8:00 a.m., a record in which the time zone column managed in the counting management DB 7003 includes 2019-04-16 08 is identified.

[0268] Next, the storage / readout unit 79 extracts moving area information (left, up, right, down) of an object such as a person from the data in the display area management DB 6001 received in step S502 from the distributed data processing terminal 6, using the area number in the record searched in step S504-3 as a search key (S504-6). Then, if the moving area is "left" (S504-7; left), the counting unit 76 counts up ("+1") to indicate a move to the left (S504-8). If the moving area is "up" (S504-7; up), the counting unit 76 counts up ("+1") to indicate a move to the up (S504-9). If the moving area is "right" (S504-7; right), the calculation unit 65 counts up ("+1") to indicate a move to the right (S504-10). In this example, the area below the attention area is outside the store, so it is not counted up. After the processes of steps S504-8, 9, and 10, the process returns to the process of step S504-1.

[0269] With this, the processing shown in FIG. 47 is completed.

[0270] 46, the transmitter / receiver 71 of the centralized data processing server 7 transmits movement information to the distributed data processing terminal 6 (S505). This movement information includes count data, which is data on the result of the counting in step S504.

[0271] Next, in the distributed data processing terminal 6, the display control unit 67 displays the movement information received in step S505 on the display 517 in a movement result display screen represented by a bar graph as shown in Fig. 48 (S506). Fig. 48 shows an example of a display in which the counting results are graphed. Each bar graph shows the counting results for each direction stacked up. By displaying the graph for each unit time in this way, it is possible to visualize in which direction the person has moved. In the example of FIG. 48, since it is the current day, the time beyond the time when the movement information display button e1 is pressed (for example, after 20:00) is not displayed. Also, the arrow arranged to the right of the date at the top of the graph can display the results of the next day, but since it is the current day in FIG. 48, it is indicated by a dimmed display that there is no reaction even if it is pressed. Here, if the arrow to the left of the date is pressed, the graph of the previous day is updated and displayed according to the process of FIG. 46. In this case, the results of all 24 hours are displayed, and the arrow to the right of the date is displayed in a dark color, so that the results of the day can be redisplayed. In addition, when the reception unit 62 receives the pressing of the "OK" button f1 at the bottom right of FIG. 48, the display control unit 67 switches from the movement result display screen of FIG. 48 to the collation result display screen of FIG. 45.

[0272] [Main Effects of the Embodiments] As described above, according to the present embodiment, by setting a predetermined area in a single omnidirectional image, the predetermined area image, which is an image of the predetermined area, can be set as a detectable range in the omnidirectional image in which an object can be detected. That is, the predetermined area image and the detectable range image are the same. This provides an effect that a desired predetermined area in a captured image can be easily set as a detectable range for an object.

[0273] The distributed processing system 100 also transmits partial image data, which is a region of a specific object, to the centralized data processing server 7 (S313), receives an object ID for identifying the assigned object from the centralized data processing server 7 as a result of the centralized data processing server 7 checking (S315), and transmits to the centralized data processing server 7 a first region number for identifying a first region among a plurality of regions included in the region captured by a single imaging unit 40, and an object ID of an object present in the first region, in association with each other (S304). This provides an effect that a specific object can be detected even if it is divided into a plurality of regions by a single imaging unit. Furthermore, by repeating the above-mentioned transmission, an effect that a specific object can be detected and tracked using a single imaging unit even if the specific object moves and the movement of the specific object spans a plurality of regions is provided.

[0274] Moreover, when the same object is photographed by the real-time data processing terminal 3, the photographed image data and a matching request are not transmitted from the real-time data processing terminal 3 to the centralized data processing server 7 via the near-terminal data processing device 5 and the distributed data processing terminal 6. This has the effect of preventing the user of the real-time data processing terminal 3 from being charged more communication fees than necessary.

[0275] 〔supplement〕 When changing areas 2, 3, and 4, if the area is being changed so that its positional relationship with respect to the attention area "1" deviates from the positional relationship set on the area setting screen in Fig. 30, the display control unit 67 can display a warning to prevent erroneous changes. For example, when the areas are set as shown in Fig. 30 and the image of area 2 is changed, a semi-transparent red background color may be superimposed on the screen while the area is being changed to the right of the attention area.

[0276] In the above embodiment, in steps S217 and S218, only the distance between the latest object detection position and the past object detection position is used as the criterion for judgment, but this is not limited thereto. For example, the area difference between the latest object partial image and the past object partial image may also be used as the criterion for judgment. Specifically, when the following two conditions (Condition 1) and (Condition 2) are satisfied, the process may proceed to step S219. According to this judgment, even if the current object detection position is close to the past object detection position, if there is an unnatural difference in the size of the detection area, it is possible not to recognize them as the same object (here, a person). This can further reduce transmissions (S115, S121) that should not require a matching request. (Condition 1) The distance between the past object and the latest object is less than (or equal to) a predetermined threshold (same as step S218). (Condition 2) The absolute value of the area difference between the past object and the latest object is less than (or equal to) a predetermined threshold value. The area difference in condition 2 is the difference between the areas (width*height) calculated by the calculation unit 31 from the height and width of the partial images.

[0277] In the above embodiment, the standby state mark m1 is represented by "!", but this is not limited to this. For example, it may be an hourglass icon or a comment such as "Verifying" or "Authenticating".

[0278] Also, although the captured image data has been described, the present invention is not limited to this, and may be created image data created by the user without being captured. In this case, the captured image data and the created image data are examples of "image data". Also, the image acquisition terminal 2 is an example of a communication terminal, and image data may be created on the own terminal without being acquired from an external source. Furthermore, the communication terminal may acquire sound data by collecting sound, may acquire temperature data by a temperature sensor, or may acquire humidity data by a humidity sensor.

[0279] The image data described above may be the captured image data itself, or partial image data representing a part of an object included in the captured image data (e.g., a person's face) or the whole of an object (e.g., a person's whole body). Furthermore, the image data may be the created image data itself, or partial image data representing a part of a character such as a two-dimensional character or a three-dimensional character included in the created image data, or the whole of a character.

[0280] Furthermore, each component such as the CPU 301, 501, 701 may be a single component or a plurality of components. Furthermore, there may be a plurality of image acquisition terminals 2, distributed data processing terminals 6, and centralized data processing servers 7. Furthermore, the distributed data processing terminal 6 may be a server.

[0281] Each function in the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment can refer to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or an ASIC (Application Specific Integrated Circuit) designed to execute each of the above-described functions. This includes devices such as digital signal processors (DSPs), field programmable gate arrays (FPGAs), systems on a chip (SOCs), GPUs, and conventional circuit modules. [Explanation of symbols]

[0282] 2. Image acquisition terminal (an example of a communication terminal) 3 Real-time data processing terminal 5 Near-end data processing device 6 Distributed data processing terminal (an example of a setting device, an example of a display device) 7 Centralized data processing server (an example of a management device, an example of other devices) 31 Calculation section 34 Image processing unit (an example of an image processing means) 35 Object detection unit (an example of an object detection means) 38 Connection section (an example of acquisition means) 51 Transmitter / receiver 61 Transmitting / receiving unit (an example of a receiving means, an example of a transmitting means) 62 Reception 65 Calculation unit (an example of a counting means) 67 Display control unit (an example of a display control means) 69 Memory / readout section 71 Transmitter / receiver 74 Feature Generation Unit 75 Matching section 100 Distributed processing system (an example of a processing system) 200 Intranet 517 Display (an example of a display means) 600 Internet 3001 Image sensor information management DB 3002 Cycle Value Management DB 3003 Image acquisition program DB 3004 Synthesis processing program management DB 3005 Distortion correction program management DB 3006 Service Program Management DB 3007 Object information management DB 3008 Object Displacement Management DB 6000 storage section 7001 Matching data management DB 8000 storage section 8001 Session Management DB 8002 Authentication server management DB 9000 storage section 9001 Authentication Management DB [Prior art documents] [Patent documents]

[0283] [Patent Document 1] JP 2006-113711 A

Claims

1. a transmission means for transmitting image data representing a predetermined object to a management device that performs image matching; a receiving means for receiving from the management device object identification information for identifying the object that is assigned as a result of the collation by the management device; a display control means for simultaneously displaying a plurality of predetermined area images, which are images of predetermined areas among a plurality of areas included in a captured image obtained by capturing an image using a single imaging unit; a receiving means for receiving a setting of each of a plurality of predetermined area images as a detectable range in which an object can be detected in the captured image; having the detectable range includes a collation range in which an image of an object can be collated; the collation range is a predetermined area within the detectable range, the transmitting means transmits, to the management device, first area identification information for identifying a first area among the plurality of areas and the object identification information of the object present in the first area in association with each other. A processing system comprising:

2. the size of the predetermined area of ​​the collation range is set by a margin set inside the detectable range; 2. The processing system according to claim 1.

3. The processing system described in claim 1 or 2, characterized in that when the object moves from the first area to the second area, the transmission means associates second area identification information for identifying the second area with the object identification information of the object present in the second area and transmits them to the management device.

4. 4. The processing system according to claim 1, The display control means causes the display means to display, for each moving direction, a number counted by a counting means that counts the number of movements of the object for each moving direction of the object at a predetermined time interval. A processing system having

5. 5. The processing system according to claim 1, wherein the image data is partial image data representing a portion of the object.

6. 6. The processing system according to claim 1, wherein each of the plurality of regions is a predetermined region among the plurality of regions obtained from the omnidirectional image captured by the single imaging unit.

7. A processing system according to any one of claims 1 to 5; The management device; A communication system having The processing system includes: A transmitting means for transmitting image data representing a predetermined object to a management device which performs image matching; a receiving means for receiving from the management device object identification information for identifying the object that is assigned as a result of the comparison by the management device; a display control means for simultaneously displaying a plurality of predetermined area images, which are images of predetermined areas among a plurality of areas included in a captured image obtained by capturing an image using a single imaging unit; a receiving means for receiving a setting of each of a plurality of predetermined area images as a detectable range in which an object can be detected in the captured image; having the detectable range includes a collation range in which an image of an object can be collated; the collation range is within a predetermined area of ​​the detectable range, A communication system characterized in that the transmission means associates first area identification information for identifying a first area among the multiple areas with the object identification information of the object present in the first area and transmits the associated information to the management device.

8. A processing method executed by a processing system that performs a predetermined process on a management device that performs image matching, comprising: a transmitting step of transmitting image data representing a predetermined object to the management device; a receiving step of receiving, from the management device, object identification information for identifying the object that has been assigned as a result of the matching by the management device; a display step of simultaneously displaying a plurality of predetermined area images, which are images of predetermined areas among a plurality of areas included in a captured image obtained by capturing an image using a single imaging unit; a receiving step of receiving a setting of each of a plurality of predetermined area images as a detectable range in which an object can be detected in the captured image; having the detectable range includes a collation range in which an image of an object can be collated; the collation range is a predetermined area within the detectable range, A processing method characterized in that the transmission step includes a process of associating first area information for identifying a first area among the multiple areas with the object identification information of the object present in the first area and transmitting the associated information to the management device.

Images