Distributed processing system, communication system, image acquisition terminal, and distributed processing method
The distributed processing system optimizes communication terminals by transmitting partial image data only when certain detection criteria are met, reducing unnecessary processing requests and communication load.
Patent Information
- Application Number
- JP2023124149
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-03-31
- Filing Date
- 2023-07-31
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2039-03-13
AI Technical Summary
Conventional communication terminals send unnecessary recognition requests when detecting the same object multiple times, leading to excessive processing requests.
A distributed processing system with an image acquisition terminal and data processing terminal that detects objects and transmits partial image data only when a predetermined number of detections is exceeded, using a data processing server for matching and displaying results.
This approach suppresses unnecessary processing requests and reduces communication load by transmitting partial image data only when specific conditions are met, optimizing resource usage.
Smart Images

Figure 0007582393000001 
Figure 0007582393000002 
Figure 0007582393000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a communication terminal, a communication system, a communication method, and a program. [Background technology]
[0002] In recent years, one of the revenue models for Web APIs (Application Programming Interfaces) is the "billing model," which charges a fee when the Web API is used. Although the unit price needs to be set low, this model is already known as one of the mainstream models because a certain amount of revenue can be expected as the number of uses increases.
[0003] The disclosure includes a step in which a service providing server sends to a control device an instruction to change specific parameters that determine the operation of the control device, for the purpose of a provider of an information service charging a user for changing various parameters that determine the operation of the terminal device owned by the user, a step in which the control device changes the specific parameters only when it receives an instruction to change the specific parameters from the service providing server, and a step in which, when the service providing server sends an instruction to change the specific parameters to the control device, it charges the user of the control device an amount equivalent to the instruction (see Patent Document 1). Summary of the Invention [Problem to be solved by the invention]
[0004] However, in conventional communication terminals, every time an object (such as a face or person) is detected, a recognition request is sent to the server providing the service. Therefore, if the same object has been recognized before, a recognition request is sent again, which can result in unnecessary processing requests being executed depending on the application. [Means for solving the problem]
[0005] The invention according to claim 1 is a distributed processing system having an image acquisition terminal that acquires an image, and a data processing terminal that controls processing related to matching based on the image, wherein the image acquisition terminal includes an image acquisition means that acquires image data relating to an image in which at least one object is captured, an object detection means that detects at least one object from the image data acquired by the image acquisition means, and a transmission means that, when the number of times a first object and a second object are detected from the image data by the object detection means exceeds a predetermined value, transmits to the data processing terminal first partial image data based on the first object and second partial image data based on the second object in the image data. and the data processing terminal comprises: an image receiving means for receiving the first partial image data and the second partial image data transmitted by the transmitting means; a requesting means for requesting a data processing server capable of communicating via a network to perform a matching process based on the first partial image data and the second partial image data received by the image receiving means; a matching result receiving means for receiving a result of the matching based on the first partial image data and the second partial image data processed by the data processing server; and a display control means for displaying the result of the matching received by the matching result receiving means on a display device. Effect of the Invention
[0006] As described above, according to the present invention, it is possible to suppress unnecessary processing requests. [Brief description of the drawings]
[0007] [Figure 1] 1 is a schematic diagram of a communication system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a hardware configuration diagram of a real-time data processing terminal. [Diagram 3] FIG. 2 is a hardware configuration diagram of an imaging unit. [Figure 4]Figure 4(a) shows a hemispherical image (front side) captured by imaging unit 40b, Figure 4(b) shows a hemispherical image (rear side) captured by imaging unit 40b, and Figure 4(c) shows an image represented using equirectangular projection. [Diagram 5] FIG. 2 is a hardware configuration diagram of a near-terminal data processing device or a distributed data processing terminal. [Figure 6] FIG. 2 is a hardware configuration diagram of a centralized data processing server. [Figure 7] FIG. 2 is a software configuration diagram of a real-time data processing terminal and a near-terminal data processing device in the image acquisition terminal. [Figure 8] FIG. 1 is a functional block diagram of a communication system. [Figure 9] FIG. 1 is a functional block diagram of a communication system. [Figure 10] 1A is a conceptual diagram of an imaging element information management table, and FIG. 1B is a conceptual diagram of a cycle value management table. [Figure 11] 1A is a conceptual diagram of an image acquisition program management table, FIG. 1B is a conceptual diagram of a synthesis processing program management table, FIG. 1C is a conceptual diagram of a distortion correction program management table, and FIG. 1D is a conceptual diagram of a service program management table. [Figure 12] FIG. 13 is a conceptual diagram of a collation data management table. [Figure 13] FIG. 2 is a conceptual diagram of a session management table. [Figure 14] FIG. 1 is a conceptual diagram of a terminal ID. [Figure 15] FIG. 4 is a conceptual diagram of an authentication server management table. [Figure 16] FIG. 4 is a conceptual diagram of an authentication management table. [Figure 17] FIG. 11 is a sequence diagram showing an authentication process. [Figure 18] FIG. 11 is a sequence diagram showing an authentication process. [Figure 19] FIG. 13 is a diagram showing an example of a screen of a distributed data processing terminal. [Figure 20] FIG. 13 is a diagram showing an example of a screen of a distributed data processing terminal. [Figure 21]FIG. 11 is a sequence diagram showing processing of a request to start image recognition. [Figure 22] FIG. 11 is a sequence diagram showing preparation processing for real-time processing of the real-time data processing terminal. [Diagram 23] FIG. 11 is a sequence diagram showing a program acquisition process. [Figure 24] FIG. 11 is a sequence diagram showing an image recognition process. [Diagram 25] 11 is a flowchart showing object detection processing in real-time processing. [Figure 26] 11 is a flowchart showing a process of generating an event in real-time processing. [Figure 27] 11A and 11B are diagrams showing examples of display of captured images in the distributed data processing terminal; [Figure 28] FIG. 11 is a sequence diagram showing a matching process. [Figure 29] FIG. 13 is a diagram showing a capture range of an imaging unit when an image acquisition terminal is installed in a room. [Diagram 30] FIG. 29 is a sequence diagram showing another example of the matching process shown in FIG. 28 according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] First embodiment [Overall Overview] Hereinafter, an overall outline of the first embodiment of the present invention will be described with reference to the drawings. This embodiment discloses a communication system that realizes edge computing. That is, this embodiment discloses an invention that realizes a service provided by the service providing server 8 at the image acquisition terminal 2 through cooperation between processing at the service providing server 8 and processing at the image acquisition terminal 2.
[0009] [Overall configuration of the embodiment] Fig. 1 is a schematic diagram of a communication system according to this embodiment. As shown in Fig. 1, the communication system according to this embodiment is constructed by a real-time data processing terminal 3, a near-terminal data processing device 5, a distributed data processing terminal 6, a centralized data processing server 7, a service providing server 8, and authentication servers 9a, 9b, and 9c. The real-time data processing terminal 3, the near-terminal data processing device 5, and the distributed data processing terminal 6 construct a distributed processing system 100. Although there are a plurality of distributed processing systems 100, only one is shown in Fig. 1 to simplify the drawing.
[0010] The near-terminal data processing device 5 is communicatively connected to a distributed data processing terminal 6 via an intranet 200. The distributed data processing terminal 6 is communicatively connected to a centralized data processing server 7, a service providing server 8, and authentication servers 9a, 9b, and 9c via the Internet 600. Note that the authentication server 9 is a collective term for the authentication servers 9a, 9b, and 9c.
[0011] Among these, the real-time data processing terminal 3 is a terminal that performs real-time processing to obtain captured image data. An imaging unit 40 having an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor for capturing an image of a subject is detachably connected to the real-time data processing terminal 3. As a result, the real-time data processing terminal 3 digitizes the captured image data (hereinafter referred to as "captured image data") input from the imaging unit 40, and detects an image of an object (here, a face) in real time (for example, every 1 / 60 seconds). The real-time data processing terminal 3 transmits data of a partial image (hereinafter referred to as "partial image data"), which is a region of the object part in the captured image, to the near-terminal data processing device 5.
[0012] The near-terminal data processing device 5 is disposed closest to the real-time data processing terminal 3, and is closely connected one-to-one with the real-time data processing terminal 3 via a data bus or a USB (Universal Serial Bus), for example, in terms of hardware. The near-terminal data processing device 5 encodes the partial image data received from the real-time data processing terminal 3 into a general-purpose format such as JPEG (Joint Photographic Experts Group), and then transmits the encoded data to the distributed data processing terminal 6 via the intranet 200 as data to be matched when face image matching is performed. Usually, the real-time data processing terminal 3 and the near-terminal data processing device 5 are connected together, and here, the real-time data processing terminal 3 and the near-terminal data processing device 5 constitute the image acquisition terminal 2.
[0013] The distributed data processing terminal 6 is a computer that is used by a user at a location relatively close to the near-terminal data processing device 5 and can accept various operations from the user. The distributed data processing terminal 6 registers and stores in advance matching data when face image matching is performed. The distributed data processing terminal 6 can request the centralized data processing server 7 to match the matching data with the data to be matched via the Internet 600. In this case, the distributed data processing terminal 6 transmits the data to be matched received from the near-terminal data processing device 5 and the matching data registered in advance. Furthermore, the distributed data processing terminal 6 can receive response information including the similarity as a response from the centralized data processing server 7. The distributed data processing terminal 6 can also display the matching result via a graphic interface. The response information may include the name of an object derived from the similarity in addition to the similarity. The name of the object is, for example, the name of an automobile, an airplane, a train, or an object, a person's name, a model number, an identification code, or the like.
[0014] The centralized data processing server 7 is installed at a location relatively far from the near-terminal data processing device 5, and can communicate with the distributed data processing terminals 6 via a communication network such as the Internet 600. When the centralized data processing server 7 receives matching request information indicating a matching request, as well as matching data and matched data, it matches the matching data with the matched data and determines the degree of similarity. Then, the centralized data processing server 7 transmits response information including the degree of similarity to the distributed data processing terminals 6.
[0015] The service providing server 8 is a server that provides various services to the image acquisition terminal 2.
[0016] The authentication server 9a performs ID authentication to determine whether the image acquisition terminal 2 has a valid right to receive a service from the service providing server 8. The authentication servers 9b and 9c perform the same authentication.
[0017] [Hardware configuration] Next, the hardware configuration of the communication system of this embodiment will be described with reference to FIG. 2 to FIG.
[0018] <Hardware configuration of real-time data processing terminal> 2 is a diagram showing the hardware configuration of the real-time data processing terminal 3. The real-time data processing terminal 3 includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a CMOS sensor 305, an acceleration / direction sensor 306, a media I / F 308, and a GPS receiver 309.
[0019] Among these, the CPU 301 controls the overall operation of the real-time data processing terminal 3. The ROM 302 stores programs used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The EEPROM 304 reads or writes various data such as programs for the real-time data processing terminal under the control of the CPU 301. The CMOS sensor 305 captures an image of a subject (mainly a blind spot of the imaging unit 40) under the control of the CPU 301 to obtain captured image data. The acceleration / direction sensor 306 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I / F 308 controls the reading or writing (storage) of data from or to a recording medium 307 such as a flash memory. The GPS receiver 309 receives GPS signals from GPS satellites.
[0020] The real-time data processing terminal 3 also includes an imaging unit I / F 313 , a microphone 314 , a speaker 315 , a sound input / output I / F 316 , a display 317 , an external device connection I / F 318 , and a touch panel 321 .
[0021] Among these, the imaging unit I / F 313 is a circuit that controls the driving of the imaging unit 40 when an external imaging unit 40 is connected. The microphone 314 is a type of built-in sound collecting means that inputs sound. The sound input / output I / F 316 is a circuit that processes input and output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display 317 is a type of display means such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 318 is an interface for connecting various external devices. The touch panel 321 is a type of input means that allows a user to operate the real-time data processing terminal 3 by pressing the display 317.
[0022] The real-time data processing terminal 3 also includes a bus line 310. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 301 shown in FIG.
[0023] <Hardware configuration of the imaging unit> Fig. 3 is a hardware configuration diagram of an imaging unit. In particular, Fig. 3(a) is a hardware configuration diagram of a monocular imaging unit 40a among the imaging units 40. Fig. 3(b) is a hardware configuration diagram of a compound-eye imaging unit 40b among the imaging units 40. Note that imaging unit 40 is a collective term for multiple types of imaging units (imaging units 40a, 40b, etc.) that differ in the number of imaging elements or the type of lenses.
[0024] 3(a), the imaging unit 40a has an imaging element 401a such as a CMOS or a CCD, a lens 402a, and a connection I / F 408a for electrically connecting to the imaging unit I / F 313 of the real-time data processing terminal 3. When the imaging unit 40a is connected to the imaging unit I / F 313 of the real-time data processing terminal 3, the imaging element 401a captures an image in response to an imaging control signal sent from the imaging unit I / F 313 via the connection I / F 408a, and transmits captured image data to the imaging unit I / F 313 via the connection I / F 408a.
[0025] 3B, the imaging unit 40b has imaging elements 401b1 and 401b2 such as CMOS or CCD, lenses 402b1 and 402b2, and a connection I / F 408b for electrically connecting to the imaging unit I / F 313 of the real-time data processing terminal 3. The lenses 402b1 and 402b2 are, for example, fisheye lenses. When the imaging unit 40a is connected to the imaging unit I / F 313 of the real-time data processing terminal 3, the imaging element 401a captures an image based on an imaging control signal sent from the imaging unit I / F 313 via the connection I / F 408a, and transmits multiple captured image data to the imaging unit I / F 313 via the connection I / F 408b.
[0026] The imaging unit 40a shown in Fig. 3(a) can obtain a general planar image, while the imaging unit 40b shown in Fig. 3(b) can obtain a celestial sphere image. Here, an outline of the process from the image captured by the imaging unit 40b to the creation of an equirectangular projection image EC will be described with reference to Fig. 4. Fig. 4(a) shows a hemispherical image (front side) captured by the imaging unit 40b, Fig. 4(b) shows a hemispherical image (rear side) captured by the imaging unit 40b, and Fig. 4(c) shows an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image").
[0027] As shown in Fig. 4(a), the image obtained by the image sensor 401b1 becomes a hemispherical image (front side) curved by the lens 402b1. Also, as shown in Fig. 4(b), the image obtained by the image sensor 403b becomes a hemispherical image (rear side) curved by the lens 402b2. Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are synthesized to generate an equirectangular projection image EC as shown in Fig. 4(c).
[0028] <Hardware configuration of near-terminal data processing device and distributed data processing terminal> 5 is a hardware configuration diagram of the near-terminal data processing device or the distributed data processing terminal. Here, the near-terminal data processing device 5 and the distributed data processing terminal 6 are shown to have the same hardware configuration, so the near-terminal data processing device 5 will be described and the description of the distributed data processing terminal 6 will be omitted.
[0029] As shown in FIG. 5, the near-terminal data processing device 5 includes a CPU 501 , a ROM 502 , a RAM 503 , an EEPROM 504 , a CMOS sensor 505 , an acceleration / direction sensor 506 , a media I / F 508 , and a GPS receiving unit 509 .
[0030] Among these, the CPU 501 controls the operation of the entire near-terminal data processing device 5. The ROM 502 stores a program used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as programs for the near-terminal data processing device under the control of the CPU 501. The CMOS sensor 505 captures an object (mainly a self-portrait of the user who operates the near-terminal data processing device 5) under the control of the CPU 501 to obtain captured image data. The acceleration / direction sensor 506 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I / F 508 controls the reading or writing (storage) of data from or to a recording medium 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.
[0031] The near-terminal data processing device 5 also includes a long-distance communication circuit 511, an antenna 511a of the long-distance communication circuit 511, a CMOS sensor 512, an image sensor I / F 513, a microphone 514, a speaker 515, an audio input / output I / F 516, a display 517, an external device connection I / F 518, a short-distance communication circuit 519, an antenna 519a of the short-distance communication circuit 519, and a touch panel 521.
[0032] Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via the intranet 200. The CMOS sensor 512 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 501 to obtain captured image data. The imaging element I / F 513 is a circuit that controls the driving of the camera 512. The microphone 514 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 516 is a circuit that processes input and output of sound signals between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a type of display means such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 518 is an interface for connecting various external devices. The short-distance communication circuit 519 is a communication circuit such as NFC (Near Field radio Communication) or Bluetooth (registered trademark). The touch panel 521 is a type of input means that a user operates the near-terminal data processing device 5 by pressing the display 517.
[0033] Further, the near-terminal data processing device 5 includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501 shown in FIG.
[0034] <Hardware configuration of centralized data processing server, service providing server, and authentication server> 6 is a hardware configuration diagram of the centralized data processing server, the service providing server, or the authentication server. Here, the centralized data processing server 7, the service providing server 8, and the authentication server 9 are shown to have the same hardware configuration, so only the centralized data processing server 7 will be described, and descriptions of the service providing server 8 and the authentication server 9 will be omitted.
[0035] Fig. 6 is a hardware configuration diagram of the centralized data processing server. The centralized data processing server 7 is constructed by a computer, and as shown in Fig. 6, includes a CPU 701, a ROM 702, a RAM 703, an HD 704, an HDD (Hard Disk Drive) 705, a recording medium 706, a media I / F 707, a display 708, a network I / F 709, a keyboard 711, a mouse 712, a CD-RW drive 714, and a bus line 710. Since the centralized data processing server 7 functions as a server, it does not need to have input devices such as the keyboard 711 and the mouse 712, or output devices such as the display 708.
[0036] Of these, the CPU 701 controls the overall operation of the centralized data processing server 7. The ROM 702 stores programs used to drive the CPU 701. The RAM 703 is used as a work area for the CPU 701. The HD 704 stores various data such as programs. The HDD 705 controls reading or writing of various data from the HD 704 under the control of the CPU 701. The media I / F 707 controls reading or writing (storing) of data from a recording medium 706 such as a flash memory. The display 708 displays various information such as a cursor, a menu, a window, characters, or an image. The network I / F 709 is an interface for data communication using the Internet 600. The keyboard 711 is a type of input means having multiple keys for inputting characters, numerical values, various instructions, and the like. The mouse 712 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, and the like. A CD-RW drive 714 controls the reading of various data from a CD-RW (Compact Disc-ReWritable) 713 as an example of a removable recording medium.
[0037] 6. The centralized data processing server 7 also includes a bus line 710. The bus line 710 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 701 shown in FIG.
[0038] [Software configuration] FIG. 7 is a software configuration diagram of the real-time data processing terminal and the near-terminal data processing device in the image acquisition terminal.
[0039] 7, the real-time data processing terminal 3 has an OS 300 and an image recognition application (hereinafter, "application" will be referred to as "app") AP1. The image recognition application AP1 runs on the RAM 303 of the real-time data processing terminal 3. Of these, the OS 300 is basic software that provides basic functions and manages the entire real-time data processing terminal 3. The image recognition application AP1 is an application for recognizing the faces of people, animals, etc. from captured images.
[0040] Further, the near terminal data processing device 5 has an OS 500 and a communication application AP2. The communication application AP2 runs on the RAM 503 of the near terminal data processing device 5. Of these, the OS 500 is basic software that provides basic functions and manages the entire near terminal data processing device 5. The communication application AP2 is an application for communicating with other terminals (devices) such as the distributed data processing terminal 6.
[0041] In this way, in the image acquisition terminal 2, the real-time data processing terminal 3 performs image recognition, while the near-terminal data processing device 5 communicates with the distributed data processing terminal 6 via the intranet 200, thereby enabling distributed processing to be performed.
[0042] The real-time data processing terminal 3 and the near-terminal data processing device 5 are equipped with not only an OS but also a driver, a software development kit (SDK), or an application programming interface (API), which may be different from each other.
[0043] Next, the functional configuration and processing of an embodiment of the present invention will be described with reference to the drawings.
[0044] [Functional configuration] First, the functional configurations of the terminals, devices, and servers constituting the communication system according to the present embodiment will be described with reference to Fig. 8 to Fig. 16. Fig. 8 is a functional block diagram of the communication system according to the embodiment. Fig. 8 particularly shows the functional blocks of the image acquisition terminal 2.
[0045] <Functional configuration of real-time data processing terminal> 8, the real-time data processing terminal 3 has a determination unit 33, an image processing unit 34, an object detection unit 35, an event generation unit 36, a display control unit 37, a connection unit 38, a storage / readout unit 39, and a communication unit 48. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 2 by an instruction from the CPU 301 in accordance with the program loaded from the EEPROM 304 onto the RAM 303.
[0046] 2, the real-time data processing terminal 3 has a storage unit 3000 constructed by the ROM 302, RAM 303, and EEPROM 304. Shape model data, which will be described later, is stored in the storage unit 3000. Furthermore, the storage unit 3000 has an image sensor information management DB 3001, a cycle value management DB 3002, an image acquisition program management DB 3003, a synthesis processing program management DB 3004, a distortion correction program management DB 3005, and a service program management DB 3006 constructed therein.
[0047] The imaging element information management DB 3001 is made up of an imaging element information management table, which will be described later. The cycle value management DB 3002 is made up of a cycle value management table, which will be described later. The image acquisition program management DB 3003 is made up of an image acquisition program management table, which will be described later. The synthesis processing program management DB 3004 is made up of a synthesis processing program management table, which will be described later. The distortion correction program management DB 3005 is made up of a distortion correction program management table, which will be described later. The service program management DB 3006 is made up of a service program management table, which will be described later.
[0048] (Image sensor information management table) 10(a) is a conceptual diagram showing an imaging element information management table. In this imaging element information management table, the model number of the imaging unit 40, the number of imaging elements included in the imaging unit 40, and the type of lens used in the imaging elements are managed in association with each other. The model number is an example of type information indicating the type of imaging unit according to the number of imaging elements and the type of lens used in the imaging elements. In addition, as long as the number of imaging elements and the type of lens can be specified, the information is not limited to the model number, and may be identification information such as a product number or a barcode.
[0049] (Cycle value management table) 10B is a conceptual diagram showing a cycle value management table, which manages the number of imaging elements included in the imaging unit 40 and a cycle value (frames per second) of object recognition processing, which will be described later, in association with each other.
[0050] Each table shown in FIG. 11 is a table for managing whether or not each program is installed in the real-time data processing terminal 3.
[0051] (Image acquisition program management table) 11(a) is a conceptual diagram showing an image acquisition program management table. In this image acquisition program management table, the number of imaging elements that the imaging unit 40 has and the image acquisition program (or the name thereof) are managed in association with each other. For example, when an image acquisition program for the number of imaging elements "1" is installed in the real-time data processing terminal 3, the program "ProgC01 (image acquisition of one system)" is managed in association with each other. When an image acquisition program for the number of imaging elements "2" is installed in the real-time data processing terminal 3, the program "ProgC02 (image acquisition of two systems)" is managed in association with each other.
[0052] (Synthesis processing program management table) 11B is a conceptual diagram showing a compositing process program management table, which manages the number of image pickup elements included in the imaging unit 40 and the names of compositing processes in association with each other.
[0053] (Distortion correction program management table) 11(c) is a conceptual diagram showing a distortion correction program management table. In this distortion correction program management table, the type of lens that the imaging unit 40 has and the distortion correction program (or the name thereof) are managed in association with each other. For example, when a distortion correction program for the lens type "wide-angle" is installed in the real-time data processing terminal 3, the program "ProgW01 (wide-angle distortion correction)" is managed in association with each other. When a distortion correction program for the lens type "fisheye" is installed in the real-time data processing terminal 3, the program "ProgW02 (fisheye distortion correction)" is managed in association with each other.
[0054] (Service program management table) 11(d) is a conceptual diagram showing a service program management table. In this service program management table, an authentication server ID for identifying an authentication server and the name of a service program for executing a service provided by the image acquisition terminal 2 are managed in association with each other. For example, when a service program for authentication server ID "a01" is installed in the real-time data processing terminal 3, a program "ProgD01 (object detection)" is managed in association with each other. Also, when a service program for authentication server ID "a02" is installed in the real-time data processing terminal 3, a program "ProgD02 (object counting)" is managed in association with each other.
[0055] (Functional configuration of real-time data processing terminal) Next, each functional configuration of the real-time data processing terminal 3 will be described in more detail with reference to FIG.
[0056] The determination unit 33 of the real-time data processing terminal 3 is realized by the processing of the CPU 301, and performs various determinations. For example, the determination unit 33 determines the number of imaging elements based on the model number sent from the imaging unit 40, with reference to the imaging element information management DB 3001.
[0057] The image processing unit 34 is realized by the processing of the CPU 301, and performs various image processing by executing each program (image acquisition program, image synthesis program, distortion correction program, and service program) shown in Fig. 11. Specifically, the image processing unit 34 performs first image processing (e.g., image acquisition, image synthesis, distortion correction) on image data by executing a first program (e.g., image acquisition program, image synthesis program, and distortion correction program) that does not require authentication to be acquired. Also, the image processing unit 34 performs second image processing (e.g., object detection processing, object counting processing) on image data by executing a second program (e.g., service program) that requires authentication to be acquired.
[0058] The object detection unit 35 is realized by processing of the CPU 301, and detects feature points that are candidates for a predetermined specific object such as a face by image processing in the captured image data acquired by the image processing unit 34, and detects the position of the object in the captured image by referring to shape model data that indicates a shape model of the predetermined specific object such as a face.
[0059] Event generation unit 36 is realized by processing of CPU 301, and generates detection information (event information) indicating that object detection unit 35 has detected the position of an object.
[0060] The display control unit 37 is realized by the processing of the CPU 301, and causes a display 317, which is an example of a display means, to display various screens.
[0061] The connection unit 38 is implemented by the processing of the imaging unit I / F 313 and the CPU 301, and is an interface for mechanically and electrically connecting the imaging unit 40 to the real-time data processing terminal 3.
[0062] The storage / reading unit 39 is realized by the processing of the CPU 301 , and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000 .
[0063] The communication unit 48 is realized by the processing of the external device connection I / F 318 and the CPU 301, and transmits and receives various data (or information) to and from a communication unit 58 (described later) of the near-terminal data processing device 5 via one-to-one communication. This communication may be wireless as well as wired.
[0064] <Functional configuration of near-terminal data processing device> 8, the near-terminal data processing device 5 has a transmission / reception unit 51, a data detection unit 56, a display control unit 57, a communication unit 58, and a storage / readout unit 59. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 5 in the near-terminal data processing device 5 by an instruction from the CPU 501 according to a program loaded from the EEPROM 504 onto the RAM 503.
[0065] The near-terminal data processing device 5 also has a storage unit 5000 constructed by the ROM 502, RAM 503, and EEPROM 504 shown in FIG.
[0066] (Functional configuration of near-terminal data processing device) Next, each functional configuration of the near terminal data processing device 5 will be described in more detail with reference to FIG.
[0067] The transmitter / receiver unit 51 of the near-terminal data processing device 5 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501, and transmits and receives various data (or information) with the distributed data processing terminal 6 via a communication network (here, the intranet 200).
[0068] The data detection unit 56 is realized by the processing of the CPU 501, and detects whether or not an event that requires reception of data from the real-time data processing terminal 3 has occurred, and whether or not reception of the data has been completed.
[0069] A display control unit 57 is realized by the processing of the CPU 501, and causes a display 517, which is an example of a display means, to display various screens.
[0070] The communication unit 58 is realized by the processing of the external device connection I / F 518 and the CPU 501, and transmits and receives various data (or information) to and from the communication unit 48 of the real-time data processing terminal 3 through one-to-one communication. This communication may be wireless or wired.
[0071] The storage / reading unit 59 is realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000 .
[0072] <Functional configuration of distributed data processing terminal> 9, the distributed data processing terminal 6 has a transmitting / receiving unit 61, a receiving unit 62, a determining unit 63, a display control unit 67, and a storing / reading unit 69. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 5 in the distributed data processing terminal 6 by an instruction from the CPU 501 in accordance with the program loaded from the EEPROM 504 onto the RAM 503.
[0073] Moreover, the distributed data processing terminal 6 has a storage unit 6000 constructed by the ROM 502, RAM 503, and EEPROM 504 shown in Fig. 5. The storage unit 6000 stores collation data. Furthermore, the storage unit 6000 has a collation data management DB 6001 constructed therein. The collation data management DB 6001 is constructed by a collation data management table, which will be described later. The collation data may be stored in a data management server other than the distributed data processing terminal 6.
[0074] (Matching data management table) FIG. 12 is a conceptual diagram showing a matching data management table. In this matching data management table, an image file showing the file name of the matching data and the name of a person shown in the matching image are managed in association with each other. In addition to the conceptual diagram of the matching data management table shown here, three items may be set: a file name of the matching data, a matching result, and a matching target name. Specifically, the file name of two pieces of data for which a matching request has been made or a file name of one piece of data shown in other embodiments is set as the file name of the matching data. In the matching result item, a matching result of "OK" or "NG" that is a judgment result based on response information provided from the centralized data processing server is set. Furthermore, the name of the matching target is set to the name of the object of the matching result (for example, the name of a car, airplane, or train, a model number, a person's name, an item name, or an identification code) provided from the centralized data processing server.
[0075] (Functional configuration of distributed data processing terminal) The transmitting / receiving unit 61 of the distributed data processing terminal 6 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501 in the distributed data processing terminal 6, and transmits and receives various data (or information) to and from the centralized data processing server 7 via a communication network (here, the Internet 600). For example, the transmitting / receiving unit 61 transmits a request for matching between matching data and matched data to the centralized data processing server 7, and processes the matching result sent from the centralized data processing server 7.
[0076] The reception unit 62 is realized by the processing of the touch panel 521 and the CPU 501 in the distributed data processing terminal 6, and receives various operations from the user.
[0077] The determination unit 63 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and performs various determinations.
[0078] The display control unit 67 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and causes various screens to be displayed on the display 517, which is an example of a display means.
[0079] The storage and reading unit 69 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and stores various data (or information) in the storage unit 6000 and reads various data (or information) from the storage unit 6000. For example, the storage and reading unit 69 stores and registers the matching data (face image data in this case) in the storage unit 6000 in response to a registration request accepted by the acceptance unit 62. In addition to this, the storage and reading unit 69 may store and register the matching data and the data to be matched that are the detection targets in the storage unit 6000 (matching data management DB 6001).
[0080] <Functional configuration of the centralized data processing server> 9, the centralized data processing server 7 has a transmission / reception unit 71, a reception unit 72, a feature generation unit 74, a collation unit 75, and a storage / readout unit 79. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 6 by an instruction from the CPU 701 in accordance with a program loaded from the HD 704 onto the RAM 703 in the distributed data processing terminal 6.
[0081] The centralized data processing server 7 also has a storage unit 7000 constructed by a ROM 702, a RAM 703, and a HD 704 shown in Fig. 6. The storage unit 7000 stores feature data on the matching side (to be described later), and names and identification codes of objects having the feature data.
[0082] (Functional configuration of the centralized data processing server) The transmitting / receiving unit 71 of the centralized data processing server 7 is realized by the processing of the network I / F 709 and the CPU 701, and transmits and receives various data (or information) to and from the distributed data processing terminal 6 via a communication network (here, the Internet 600). The transmitting / receiving unit 71 receives a request for matching between matching data and matched data from the distributed data processing terminal 6, and transmits response information including the degree of similarity to the distributed data processing terminal 6.
[0083] The reception unit 72 is realized by the processing of the keyboard 711, the mouse 712, and the CPU 701, and receives various operations from the user.
[0084] The feature amount generating unit 74 is realized by processing of the CPU 701, and generates feature amount parameters from the matched data (partial image data) and matching data received by the transmitting / receiving unit 71.
[0085] The matching unit 75 is realized by processing of the CPU 701, and uses the features generated by the feature generation unit 74 to match the features of the matching data side with the features of the data to be matched, and calculates a score (points) indicating the degree of similarity.
[0086] The storage / reading unit 79 is realized by the processing of the CPU 701 , and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000 .
[0087] <Functional configuration of the service provider server> Next, each functional configuration of the service providing server 8 will be described in detail with reference to Fig. 9 and Fig. 13 to Fig. 15. As shown in Fig. 9, the service providing server 8 has a transmitting / receiving unit 81, a determining unit 82, an extracting unit 87, and a storing / reading unit 89. Each of these units is a function or means realized by any of the components shown in Fig. 6 operating in response to an instruction from the CPU 701 in accordance with the program for the service providing server 8 loaded from the HD 704 onto the RAM 703.
[0088] The service providing server 8 also has a storage unit 8000 constructed by the RAM 703 and HD 704 shown in Fig. 6. This storage unit 8000 stores various data sent from the distributed data processing terminal 6 or the authentication server 9. The storage unit 8000 also stores all of the programs shown in Fig. 11, and the service providing server 8 can transmit a requested program in response to a request from the real-time data processing terminal 3.
[0089] Furthermore, the storage unit 8000 has a session management DB 8001 and an authentication server management DB 8002. Of these, the session management DB 8001 is configured with a session management table, which will be described later. The authentication server management DB 8002 is configured with an authentication server management table, which will be described later. Each table will be described in detail below.
[0090] (Session management table) 13 is a conceptual diagram showing a session management table. In this session management table, a session ID for identifying a communication session to be established with a distributed data processing terminal 6 in order to provide a service to the distributed data processing terminal 6, a terminal ID for identifying the distributed data processing terminal 6, and an IP address of the distributed data processing terminal 6 of the user identified by the terminal ID are stored and managed in association with each other.
[0091] 14(a), (b), and (c) show an email address as an example of a terminal ID (identification), each of which is composed of an authentication part and a non-authentication part. The authentication part is a user ID used when authenticating by the authentication server 9. The non-authentication part is a part that is not used when authenticating by the authentication server 9.
[0092] In the first pattern shown in Fig. 14(a), the part to be authenticated is composed of the account name "asai", the host name "myhost", and the first part of the domain name "ricoo.com". On the other hand, the part not to be authenticated is composed of the last part of the domain name "theta1". In this case, the extraction unit 87 distinguishes between the part to be authenticated and the part not to be authenticated by using " / ".
[0093] Fig. 14(b) also shows the first pattern, but the non-authentication target part is different from that of Fig. 14(a). That is, the authentication server 9 authenticates the terminal ID shown in Fig. 14(a) as the same as the terminal ID shown in Fig. 14(b) because the authentication target part is the same.
[0094] The terminal ID may be a second pattern shown in Fig. 14(c). In this second pattern, the part to be authenticated is made up of the first part of the account name, "asai". On the other hand, the part not to be authenticated is made up of the latter part of the account name, "theta2", the host name, "myhost", and the domain name, "ricoo.com". In this case, the extraction unit 87 distinguishes between the part to be authenticated and the part not to be authenticated by using "+".
[0095] (Authentication server management table) 15 is a conceptual diagram showing an authentication server management table. In this authentication server management table, a URL (Uniform Resource Locator) for accessing each authentication server 9 is stored and managed in association with each authentication server ID for identifying each authentication server 9.
[0096] (Service server functional configuration) Next, each functional configuration of the service providing server 8 will be described in detail with reference to FIG.
[0097] The transmission / reception unit 81 of the service providing server 8 is mainly realized by instructions from the CPU 701 shown in FIG. 6 and the network I / F 709, and transmits and receives various data (or information) with the distributed data processing terminal 6 or the authentication server 9 via the communication network 600.
[0098] The determination unit 82 is mainly realized by an instruction from the CPU 701 shown in FIG. 6, and determines, for example, whether or not a communication session for providing a service to the distributed data processing terminal 6 has already been established.
[0099] The extracting unit 87 is mainly realized by commands from the CPU 701 shown in FIG. 6, and performs processing to extract a user ID (part to be authenticated) as shown in FIG. 14 from the terminal ID.
[0100] The storage / reading unit 89 is realized mainly by instructions from the CPU 701 shown in FIG. 6 and the HDD 705 , and stores various data in the storage unit 8000 and reads various data from the storage unit 8000 .
[0101] <Authentication server functional configuration> Next, the functional configuration of the authentication server 9 will be described in detail with reference to Fig. 9 and Fig. 11. The authentication server 9 has a transmission / reception unit 91, an authentication unit 92, and a storage / readout unit 99. Each of these units is a function or means realized by any of the components shown in Fig. 6 operating in response to an instruction from the CPU 701 in accordance with the program for the authentication server 9 loaded from the HD 704 onto the RAM 703.
[0102] 6. The authentication server 9 also has a storage unit 9000 constructed by the RAM 703 and HD 704 shown in Fig. 6. In this storage unit 9000, various data transmitted from the distributed data processing terminal 6 or the service providing server 8 is stored.
[0103] Furthermore, an authentication management DB 9001 is constructed in the storage unit 9000. The authentication management DB 9001 is composed of an authentication management table, which will be described later. This table will be described in detail below.
[0104] (Authentication Management Table) Figure 16(a) is a conceptual diagram showing an authentication management table held by authentication server 9a, Figure 16(b) is a conceptual diagram showing an authentication management table held by authentication server 9b, and Figure 16(c) is a conceptual diagram showing an authentication management table held by authentication server 9c.
[0105] In each authentication management table, a user ID (part to be authenticated) of a terminal ID and a password are stored and managed in association with each other.
[0106] (Authentication server functional configuration) Next, each functional configuration of the authentication server 9 will be described in detail with reference to FIG.
[0107] The transmission / reception unit 91 of the authentication server 9 is mainly realized by instructions from the CPU 701 shown in FIG. 6 and the network I / F 709, and transmits and receives various data (or information) with the distributed data processing terminal 6 and the service providing server 8 via the Internet 600.
[0108] The authentication unit 92 is mainly realized by commands from the CPU 701 shown in Fig. 6, and authenticates the ID of the distributed data processing terminal 6 by determining whether the image acquisition terminal 2 that has sent an authentication request has a legitimate right to receive the service. The reason why such authentication is performed is that the distributed data processing terminal 6 manages the image acquisition terminal 2.
[0109] The memory / read unit 99 is mainly realized by instructions from the CPU 701 shown in FIG. 6 and the HDD 705, and stores various data (or information) in the memory unit 9000 and reads various data (or information) from the memory unit 9000.
[0110] [Processing or Operation of the Present Embodiment] Next, the processing or operation of this embodiment will be described with reference to FIGS.
[0111] <Authentication process> First, the authentication process will be described with reference to Fig. 17 to Fig. 20. Fig. 17 and Fig. 18 are sequence diagrams showing the authentication process. Fig. 19 and Fig. 20 are diagrams showing examples of screens of a distributed data processing terminal.
[0112] 17, a request for an authentication destination selection screen is transmitted from the transmitting / receiving unit 61 of the distributed data processing terminal 6 to the service providing server 8 (step S21). This request includes the terminal ID of the distributed data processing terminal 6. At this time, the transmitting / receiving unit 61 also transmits the IP address of its own terminal. As a result, the transmitting / receiving unit 81 of the service providing server 8 receives the request for the authentication destination selection screen and the IP address of the distributed data processing terminal 6.
[0113] Next, the determination unit 82 of the service providing server 8 determines whether or not the terminal ID received in step S21 is associated with a specific session ID and managed in the session management table (see FIG. 13) (step S22). Next, a case where the terminal ID is not managed will be described.
[0114] The transmitting / receiving unit 81 of the service providing server 8 transmits the data of the authentication destination selection screen to the distributed data processing terminal 6 (step S23). As a result, the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the data of the authentication destination selection screen.
[0115] Next, the display control unit 63 of the distributed data processing terminal 6 causes the display 517 to display an authentication destination selection screen s1 as shown in FIG. 19 (step S24). FIG. 19 shows an example of a screen as the distributed data processing terminal 6. The authentication destination selection screen s1 displays an input field b1 for a terminal ID, an input field b2 for a password, and a login button b3 for making a login request (authentication request). Furthermore, the authentication destination selection screen s1 displays authentication server selection buttons a1, a2, and a3 for selecting authentication servers 9a, 9b, and 9c, respectively. For example, the authentication server selection button a1 is a button for when the user receives the service of object detection processing. The authentication server selection button a2 is a button for when the user receives the service of object counting processing.
[0116] Here, when the user inputs his / her terminal ID in the input field b1, his / her password in the input field b2, presses a desired button among the authentication server selection buttons a1, a2, a3, and presses the login button b3, the reception unit 62 receives each input and selection (step S25). Here, a case will be described in which the object detection processing is executed by the object detection processing program ProgD01 out of the image recognition processing by selecting the authentication server selection button a1.
[0117] The transmitting / receiving unit 61 transmits an authentication request for the ID (terminal ID or user ID in this case) to the service providing server 8 (step S26). This authentication request includes the terminal ID and password accepted in step S25, as well as the selection result of the authentication server 9 and the URL of the distributed data processing terminal 6. This selection result indicates the authentication server ID for identifying the authentication server 9. As a result, the transmitting / receiving unit 81 of the service providing server 8 receives the ID authentication request.
[0118] Next, the storage / reading unit 89 of the service providing server 8 searches the authentication server management table (see FIG. 15) using the authentication server ID received as the selection result in step S26 as a search key, and reads out the URL of the corresponding authentication server (step S27).
[0119] Next, the extraction unit 87 extracts only the user ID (part to be authenticated) from the terminal ID received in step S26 (step S28). Then, the transmission / reception unit 81 transmits an ID authentication request to the authentication server 9 indicated by the URL read in step S27 (step S29). This ID authentication request includes the user ID (part to be authenticated) extracted in step S28, the password received in step S26, and the URL of the distributed data processing terminal 6 received in step S26. As a result, the transmission / reception unit 71 of the authentication server 9 receives the user authentication request.
[0120] Next, the storage / read unit 99 of the authentication server 9 uses the pair of the user ID (part to be authenticated) and password received in step S29 as a search key, and the authentication unit 92 performs authentication using the result of searching the authentication management table (see FIG. 16) for the same pair of the part to be authenticated and the password (step S30). If the same pair is managed, the authentication unit 92 judges that the distributed data processing terminal 6 is a valid terminal for receiving a service from the service providing server 8, and if the same pair is not managed, it judges that the distributed data processing terminal 6 is not a valid terminal for receiving a service from the service providing server 8.
[0121] In step S28, the extraction unit 87 extracts the authentication target part from the terminal ID, but this is not limited to this. For example, the distributed data processing terminal 6 may extract the authentication target part from the terminal ID before the processing of step S26. In this case, the user ID (authentication target part) is transmitted in step S26, rather than the terminal ID. Also, the authentication server 9 may extract the authentication target part from the terminal ID after the processing of step S29. In this case, the terminal ID is transmitted in step S29, rather than the user ID (authentication target part).
[0122] 18, the authentication unit 92 of the authentication server 9 encrypts the token (transmission right) (step S41). Then, the transmitting / receiving unit 91 transmits an authentication result to the distributed data processing terminal 6 based on the URL of the distributed data processing terminal 6 received in step S29 (step S42). This authentication result indicates whether the distributed data processing terminal 6 is valid or not, and includes the token encrypted in step S41. As a result, the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the authentication result of the user. Hereinafter, a case where the user has valid authority will be described.
[0123] The transmitting / receiving unit 61 of the distributed data processing terminal 6 transmits a session establishment request to the service providing server 8 (step S43). This session establishment request includes the terminal ID and the encrypted token received in step S42. As a result, the transmitting / receiving unit 81 of the service providing server 8 receives the session establishment request.
[0124] Next, in order to confirm that the distributed data processing terminal 6 that transmitted the session establishment request has been determined to be valid in step S30, the transmitting / receiving unit 81 of the service providing server 8 transmits a token authentication request to the authentication server 9 (step S44). This token authentication request includes the encrypted token received in step S43. As a result, the transmitting / receiving unit 91 of the authentication server 9 receives the token authentication request.
[0125] Next, the authentication unit 92 decrypts the encrypted token received in step S44 (step S45). The authentication unit 92 then compares the token before encryption in step S41 with the token after decryption in step S45 to authenticate the token (step S46). The transmission / reception unit 91 of the authentication server 9 then transmits the authentication result of step S46 to the service providing server 8 (step S47). As a result, the transmission / reception unit 81 of the service providing server 8 receives the authentication result. Hereinafter, a case where the token is determined to be valid in step S46 will be described.
[0126] Next, the storage / read unit 89 of the service providing server 8 assigns a new session ID in the session management table (see FIG. 13), and manages the session ID in association with the terminal ID and IP address received in step S26 (step S48). Then, the transmission / reception unit 81 transmits data of the service providing screen to the distributed data processing terminal 6 (step S49). As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the data of the service providing screen.
[0127] Next, the display control unit 63 of the distributed data processing terminal 6 displays a service provision screen s2 as shown in Fig. 20 on the display 517 (step S50). Fig. 20 shows an example of a screen of the distributed data processing terminal 6. Here, as an example of the service provided, a remote control service for realizing remote control of the image acquisition terminal 2 from the distributed data processing terminal 6 will be described. The service provision screen s2 shown in Fig. 20 displays an input field c1 for an IP address for identifying a remote control target, and a "Start remote control" button c2.
[0128] <Preparation for image recognition> Next, a preparation process for image recognition will be described with reference to Fig. 21 to Fig. 23. Fig. 21 is a sequence diagram showing the process of a request to start image recognition.
[0129] As shown in FIG. 21, in the distributed data processing terminal 6, the reception unit 62 receives a start request for image recognition from a user (step S61). In this case, a GUI (Graphical User Interface) of the distributed data processing terminal 6 is used. As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits start request information for image recognition indicating a start request for image recognition to the near-terminal data processing device 5 (step S62). This start request information includes an authentication server ID for identifying the authentication server 9 that performed the authentication in the above-mentioned authentication process (see FIG. 17 and FIG. 18) and a session ID set in FIG. 18. As a result, the transmission / reception unit 51 of the near-terminal data processing device 5 receives the start request information for image recognition. Then, the communication unit 58 transmits the start request information for image recognition to the real-time data processing terminal 3 (step S63). As a result, the communication unit 48 of the real-time data processing terminal 3 receives the start request information for image recognition. In this way, by separating the user interface from the real-time data processing terminal 3, remote operation from the distributed data processing terminal 6 is possible.
[0130] Fig. 22 is a sequence diagram showing a preparation process for real-time processing of a real-time data processing terminal. For example, the real-time data processing terminal in the image acquisition terminal 2 that has received the start request information shown in Fig. 21 executes the sequence of Fig. 22.
[0131] 22, the connection unit 38 of the real-time data processing terminal 3 acquires the model number of the imaging unit 40 from the imaging unit 40 (step S71). In this case, the connection unit 38 requests the imaging unit 40 for the model number, and the imaging unit 40 transmits its own model number in response to this request.
[0132] Next, the storage / reading unit 39 searches the imaging element information management DB 3001 (see FIG. 10(a)) using the model number acquired in step S71 as a search key to read out the corresponding number of imaging elements and the type of lenses (step S72). Furthermore, the storage / reading unit 39 searches the cycle value management DB 3002 (see FIG. 10(b)) using the number of imaging elements read in step S72 as a search key to read out the corresponding cycle value (step S73).
[0133] Next, the storage / readout unit 39 searches for an image acquisition program managed in the image acquisition program management DB 3003 (see FIG. 11(a)) using the number of imaging elements read out in step S72 as a search key (step S74). Next, the storage / readout unit 39 searches for a synthesis processing program managed in the synthesis processing program management DB 3004 (see FIG. 11(b)) using the number of imaging elements read out in step S72 as a search key (step S75). Next, the storage / readout unit 39 searches for a distortion correction program managed in the distortion correction program management DB 3005 (see FIG. 11(c)) using the type of lens read out in step S72 as a search key (step S76). Next, the storage / readout unit 39 searches for a service program managed in the service program management DB 3006 (see FIG. 11(d)) using the type of lens read out in step S72 as a search key (step S77).
[0134] Next, based on the search results of steps S74 to S77, the determination unit 33 determines whether all of the programs to be executed have been installed, depending on whether the programs to be executed are managed (step S78). For example, in step S74, the storage / readout unit 39 searches for the image processing programs, and if the image processing programs are managed, the determination unit 33 determines that the programs are installed, whereas if the image processing programs are not managed, the determination unit 33 determines that the programs are not installed.
[0135] Then, when the judgment unit 33 judges that all four programs are managed (step S78; YES), the process shown in Fig. 22 ends. On the other hand, when the judgment unit 33 judges that at least one of the four programs is not installed (step S78; NO), the process proceeds to step S91 described later.
[0136] Fig. 23 is a sequence diagram showing a program acquisition process, in which the real-time data processing terminal 3 acquires, from the service providing server 8, a program that has been found not to be installed by the process shown in Fig. 22.
[0137] 23, the communication unit 48 of the real-time data processing terminal 3 transmits request information indicating a request for a program that is not installed to the communication unit 58 of the near-terminal data processing device 5 (step S91). This request information includes the name of the requested program, the authentication server ID, and the session ID.
[0138] Next, the transmitting / receiving unit 51 of the near-terminal data processing device 5 transmits the request information received by the communication unit 58 to the transmitting / receiving unit 61 of the distributed data processing terminal 6 (step S92). Then, the transmitting / receiving unit 61 of the distributed data processing terminal 6 transmits the request information to the transmitting / receiving unit 81 of the service providing server 8 (step S93).
[0139] Next, in the service providing server 8, the storage / reading unit 89 confirms the validity of the request based on the authentication server ID and the session ID, and reads out the program indicated by the program name included in the request information (step S94). Then, the transmitting / receiving unit 81 transmits the read out program to the transmitting / receiving unit 61 of the distributed data processing terminal 6 (step S95). At this time, the program name is also transmitted.
[0140] Next, the transmitting / receiving unit 61 of the distributed data processing terminal 6 transmits the program including the program name to the transmitting / receiving unit 51 of the near terminal data processing device 5 (step S96). Then, the communication unit 58 of the near terminal data processing device 5 transmits the program including the program name to the communication unit 48 of the real-time data processing terminal 3 (step S97).
[0141] Next, the memory / read unit 39 of the real-time data processing terminal 3 installs the program acquired by the communication unit 48, and manages the program name to be installed and the authentication server ID in association with each other in the tables shown in FIG. 11 (step S98).
[0142] Next, the storage / readout unit 39 starts all programs necessary for the image recognition processing (step S99). As a result, the real-time data processing terminal 3 executes the started programs, thereby starting the real-time processing shown below.
[0143] <Image recognition processing> Here, a case will be described in which the "Login to object detection processing service" button a1 shown in Fig. 19 is pressed and the service providing server 8 executes the object detection processing service among the image recognition processing services. Fig. 24 is a sequence diagram showing the image recognition processing.
[0144] First, the real-time data processing terminal 3 performs real-time processing (step S111). Here, the real-time processing will be described with reference to Fig. 25 and Fig. 26. Fig. 25 is a flowchart showing the object detection processing of the real-time processing.
[0145] 25, the determination unit 33 determines whether the number of imaging elements in the connected imaging unit 40 is one (step S201). In this case, the determination unit 33 makes a determination based on the number of imaging elements read out by the process of step S72 described above. Then, if the number of imaging elements is one (step S201; YES), the image processing unit 34 sets the cycle value for repeating the process to 1 / 60 seconds (step S202). In this case, the image processing unit 34 sets the cycle value read out by the process of step S73 described above.
[0146] Next, the connection unit 38 acquires one line of captured image data from the imaging unit 40 (step S203). This captured image data is digital image data, for example, 4K image data (width 3840 pixels × height 2160 pixels). In this case, the connection unit 38 executes processing by the image acquisition program (ProgC01 (one line)) shown in FIG. 11(a).
[0147] Next, the object detection unit 35 detects object candidates in the captured image data by detecting feature points (step S204). In this case, the object detection unit 35 uses the authentication server ID included in the start request information to determine whether or not the device has an appropriate authentication server ID from the service program management table (see FIG. 11(d)), and executes the process after determining whether or not the service program (ProgD01 (object detection process)) shown in the DB can be executed. The object detection unit 35 searches for a rectangle from the edge of the captured image while comparing it with shape model data of the object stored in advance in the storage unit 3000, and selects the position of the most likely feature point of the object. For the process of step S204, a known technology such as that disclosed in "Face Recognition Technology and Its Applications: Special Issue on Elemental Technologies and Solutions Supporting Public Safety; Biometrics Authentication, NEC Technical Journal, Vol. 63, no. 3, pp. 26-30, 2010-09" can be used.
[0148] Next, the image processing unit 34 corrects the distortion of the image of the detected object (step S205). In this case, the image processing unit 34 executes the process by the image processing program (ProgW01 (wide-angle distortion correction)) shown in Fig. 11(c). In executing this process, the real-time data processing terminal 3 does not need to use the authentication server ID included in the start request information and refer to Fig. 11(d) to determine whether or not to execute the wide-angle distortion correction program.
[0149] On the other hand, in the above step S201, if the number of image pickup elements is not one (step S201; NO), the image processing unit 34 sets the cycle value for repeating the real-time processing to 1 / 30 seconds (step S206). In this case, the image processing unit 34 sets the cycle value read out by the processing of the above step S73. By setting the cycle value to 1 / 30 seconds, the cycle value is set large compared to one input, and it is possible to prevent the image synthesis processing described below from being delayed.
[0150] Next, the connection unit 38 acquires two sets of captured image data for two systems from the imaging unit 40 (step S207). The two sets of captured image data are data for hemispherical images as shown in Figs. 4(a) and (b), respectively. In this case, the connection unit 38 executes processing using the image acquisition program (ProgC02 (two systems)) shown in Fig. 11(a). As described above, there is no need to determine whether or not this processing can be executed when it is executed.
[0151] Then, the image processing unit 34 synthesizes the two captured image data to create an equirectangular projection image EC as shown in FIG. 4(c) (step S208). In this case, the image processing unit 34 executes processing by the synthesis processing program (ProgS02 (synthesis processing)) shown in FIG. 11(b). Then, the process proceeds to step S204, where the object detection unit 35 detects object candidates in the data of the equirectangular projection image EC, thereby detecting the object. In this case, as described above, the object detection unit 35 uses the authentication server ID included in the start request information to determine whether or not to execute the service program (ProgD01 (object detection processing)) shown in FIG. 11(d), and then executes the processing.
[0152] Next, in step S205, the image processing unit 34 corrects distortion of the image of the detected object. In this case, the image processing unit 34 executes the process by the image processing program (ProgW02 (fisheye distortion correction)) shown in Fig. 11(c). In executing this process, the real-time data processing terminal 3 does not need to use the authentication server ID included in the start request information to refer to the service program management table (see Fig. 11(d)) to determine whether or not to execute the fisheye distortion correction program.
[0153] As a result, as shown in Fig. 24, the communication unit 48 of the real-time data processing terminal 3 transmits the captured image data to the communication unit 58 of the near-terminal data processing device 5 (step S112). Then, the transmission / reception unit 51 of the near-terminal data processing device 5 transmits the captured image data received in step S112 to the transmission / reception unit 61 of the distributed data processing terminal 6 (step S113). As a result, the display control unit 67 of the distributed data processing terminal 6 displays the captured image as shown in Fig. 27(a) on the display 517 in real time (step S114). Figs. 27(a) and (b) are diagrams showing a display example of the captured image in the distributed data processing terminal. Steps S112 to S114 are streaming processes.
[0154] <Communication / Communication Suppression> Next, communication and communication suppression will be described with reference to Fig. 26 to Fig. 29. Fig. 26 is a flowchart showing the event generation process in real-time processing. Fig. 28 is a sequence diagram showing the collation process. Fig. 29 is a diagram showing the shooting range of an imaging unit when an image acquisition terminal is installed in a room.
[0155] First, after the object detection unit 35 sets the number of face detections (count) to 0 (step S211), if the object detection unit 35 does not detect two objects (faces in this case) by the process of step S204 described above (step S212; NO), it waits for a predetermined time (set cycle value) (step S213) and then returns to the process of step S211 again. For example, detection is performed 30 times per second.
[0156] On the other hand, if the object detection unit 35 detects two objects (faces in this case) by the process of step S204 described above (step S212; YES), the process proceeds to step S214. If either of the detected positions is not within the collation possible range Q (step S214; NO), the process proceeds to step S213.
[0157] Here, the collation possible range Q will be described with reference to FIG. 29. FIG. 29 is a diagram showing the shooting range of the imaging unit when the image acquisition terminal 2 is installed in a certain room. The detectable range P is the range in which the shooting range of the imaging unit 40 can detect an object (here, a face). This detectable range P includes a collation possible range Q in which the image of the object (here, a face) can be collated, and an unmatchable range R outside this collation possible range Q in which matching is not possible. In FIG. 29, margins a and b are predetermined values provided at the left and right ends in the x direction and the top and bottom ends in the y direction in the detectable range P. These values are stored in the service program acquired by the real-time data processing terminal 3 in step S97 shown in FIG. 23.
[0158] The face detection by the process of step S204 described above is performed over the entire detectable range P, which is the shooting range. If the object is outside the rectangular range determined by margins a and b, the object is not within the matchable range Q (i.e., within the unmatchable range R). Here, (x1, y1) indicate the coordinates of the upper left corner of the partial image data where the object was detected.
[0159] Returning to FIG. 26, if the judgment unit 33 judges that both detection positions are within the collation possible range Q (step S214; YES), the object detection unit 35 increments the detection count by one (step S215) and then proceeds to the process of step S216. For example, if an object is at a position indicated by the coordinates (x2, y2) in FIG. 29, it is judged to be within the collation possible range Q. Then, if the detection positions of the two objects are within the collation possible range Q (step S216; YES), the process proceeds to the process of step S218. Then, if the detection count does not exceed a predetermined value (for example, 10 times) (step S216; NO), the process returns to the process of step S212 after waiting for a predetermined time (step S217).
[0160] In step S216, the process proceeds to the next step S218 only if the number of detections exceeds a predetermined value. This is because, when a user has his or her face and card (a card with a photo of the user's face printed on it) read by the imaging unit 40 of the image acquisition terminal 2, as shown in Figure 27(a), if a user who does not want to undergo face authentication tries to have the card read for just a moment, or if a user holding the card passes in front of the image acquisition terminal 2, the partial image data is not sent from the near terminal data processing device 5 to the distributed data processing terminal 6, thereby suppressing an excessive number of processing requests (or number of communications or amount of communications) (first suppression).
[0161] On the other hand, if the judgment unit 33 judges that the number of detections does not exceed the predetermined value (step S216; YES), the image processing unit 34 encodes the two partial image data including the object at the detection position in a general-purpose format such as JPEG (step S218). Then, the event generating unit 36 generates an event message notifying the near terminal data processing device 5 that the partial image data will be transmitted (step S219). Specifically, the event generating unit 36 generates an event message such as "Send" and sets (resets) the number of detections to 0 (step S220), and then proceeds to the processing of step S221. As a result, the processing of step S121 shown in FIG. 28 is executed, and each of the partial image data of the user's face and the face printed on the card is transmitted from the near terminal data processing device 5 to the distributed data processing terminal 6.
[0162] Next, if the image processing unit 34 detects two objects (faces in this case) by the process of step S204 in Fig. 25 (step S221; YES), after waiting for a predetermined time (a set cycle value) (step S222), the process returns to the process of step S220 again and the number of occurrences is reset to 0. In this case, since it is considered that the user has been having the image acquisition terminal 2 read the photo on the card for a long time, the process of step S121 prevents the same user's face and each partial image data of the face printed on the card from being sent again, thereby suppressing an excessive number of processing requests (or the number of communications or the amount of communications) (second suppression).
[0163] On the other hand, if the image processing unit 34 does not detect two objects (faces in this case) by the process of step S204 described above (step S212; NO), the detection count is incremented by one (step S223) and the process proceeds to step S224. If the detection count does not exceed a predetermined value (for example, 10 times) (step S224; NO), the process returns to the process of step S221 after waiting for a predetermined time (a set cycle value) (step S225). In this case, the user may turn the photo on the card over to himself / herself for a short time using the image acquisition terminal 2, and it is considered that the same person is still within the collation possible range Q. Therefore, the process of step S221 prevents the transmission of each partial image data of the same user's face and the face printed on the card again, thereby suppressing an excessive number of processing requests, the number of communications, and the amount of communications (third suppression).
[0164] On the other hand, if the number of detections exceeds a predetermined value (for example, 10 times) (step S224; YES), the process returns to step S213.
[0165] With the above, the process of step S111 shown in Fig. 24 is completed. As a result, the communication unit 48 transmits the event message generated in step S219 and the partial image data corrected for distortion in step S205 to the communication unit 58 of the near-terminal data processing device 5 (step S115). If multiple objects (faces in this case) are detected in the process of step S204, multiple partial image data are transmitted together with one event message. Furthermore, when the communication system is used in a store or the like, the real-time data processing terminal 3 may store the event message and the partial image data, and transmit them together to the near-terminal data processing device 5 after the store closes (for example, at night).
[0166] Next, in the near-terminal data processing device 5, the data detection unit 56 detects whether or not a "Send" event message has been received by the communication unit 58 (step S116). Then, if reception of an event message is detected (step S116; YES), the communication unit 58 receives the partial image data transmitted together with the event message (step S117). Then, the storage / readout unit 59 temporarily stores the partial image data in the storage unit 5000 (step S118).
[0167] Next, the data detection unit 56 monitors whether or not the reception of the partial image data is completed (step S119). The process of this step S119 is repeated until all partial image data for one event message is received (step S119; NO). Then, when the reception of the partial image data is completed (step S119; YES), the storage / readout unit 59 reads out all partial image data sent together with one event message from the storage unit 5000 and temporarily stored in the storage unit 5000 (step S120). Thereafter, the transmission / reception unit 51 transmits all (here, two) partial image data read out in step S120 to the transmission / reception unit 61 of the distributed data processing terminal 6 via the intranet 200 (step S121). As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the two partial image data. The two partial image data are used as targets for comparing the similarity of each other later.
[0168] <Matching process> Next, the process of comparing two partial image data will be described with reference to Fig. 28. Fig. 28 is a sequence diagram showing the process of comparing.
[0169] First, when the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the two partial image data by the process of step S121 described above, it transmits matching request information indicating a matching request to the centralized data processing server 7 via the Internet 600 (step S314). This matching request information includes the two partial image data. As a result, the transmitting / receiving unit 71 of the centralized data processing server 7 receives the matching request information.
[0170] Next, in the centralized data processing server 7, the feature generating unit 74 performs conversion to decode both data (two partial image data) into bitmap data, and generates feature parameters for identifying an individual, such as the contours and inclinations of the eyes and nose of the face images related to both data (step S315). Then, the matching unit 75 compares the feature parameters related to both data to calculate the similarity (step S316). For example, a deep neural network (DNN) is used for the calculation method of this similarity, as disclosed in "Deep Learning and Image Recognition: Fundamentals and Recent Trends (<Special Feature> Neuroscience and Mathematical Modeling) Operations Research: Science of Management 60(4), 198-204, 2015-04-01". The feature parameters of the partial image data of the user are an example of the first feature parameters, and the feature parameters of the partial image data of the user's photo on the card are an example of the second feature parameters.
[0171] Next, the transmitting / receiving unit 71 of the centralized data processing server 7 transmits response information indicating a response corresponding to the matching request in step S314 to the transmitting / receiving unit 61 of the distributed data processing terminal 6 via the Internet 600 (step S317). This response information includes the similarity (e.g., 0.9) that is the calculation result in step S316. As a result, the transmitting / receiving unit 61 of the distributed data processing terminal 6 receives the response information. Note that, if the similarity shown above is greater than a predetermined threshold, the centralized data processing server 7 may read out from the storage unit 7000 the name of an object having the feature amount corresponding to the feature amount, an identification code, etc., and include it in the response information to transmit to the distributed data processing terminal 6 in step S317.
[0172] Next, the judgment unit 63 of the distributed data processing terminal 6 compares the similarity with a predetermined threshold (for example, 0.8), and judges the similarity as "OK" if it is greater than the threshold, and judges it as "NG" if it is not (step S318). Then, the display control unit 67 displays a matching result message as shown in Fig. 27(b) on the real-time captured image shown in Fig. 27(a) on the display 517 of the distributed data processing terminal 6 (step S319). This matching result message m1 displays the matching result "OK" or "NG".
[0173] [Main Effects of the Present Embodiment] As described above, according to this embodiment, only when the number of detections exceeds a predetermined value in step S216, the process proceeds to the process of generating an event message (step S219). That is, when the number of detections of the first object and the second object by the object detection unit 35 exceeds a predetermined value, the transmission / reception unit 51 transmits the first image data based on the first object and the second image data based on the second object to another device such as the distributed data processing terminal 6 (see S121). As a result, when a user has his / her face and a card (a card on which a photo of his / her face is printed) read by the imaging unit 40 of the image acquisition terminal 2 as shown in FIG. 27(a), in a case where a user who does not want to have his / her face authenticated tries to have the card read for only a moment or a user holding the card passes in front of the image acquisition terminal 2, the partial image data is not transmitted from the near terminal data processing device 5 to the distributed data processing terminal 6, and an excessive number of processing requests (or the number of communications or the amount of communications) can be suppressed (first suppression).
[0174] Also, if the image processing unit 34 detects two objects (faces in this case) (step S221; YES), it waits for a predetermined time (set cycle value) (step S222) and returns to the process of step S220 again, resetting the number of occurrences to 0. That is, after the first and second image data are transmitted by the transmitting / receiving unit 51, if the first and second objects are detected again by the object detection unit 35 without a state in which the first and second objects are not detected, the transmitting / receiving unit 51 does not transmit the first and second image data. In this case, it is considered that the user has the image acquisition terminal 2 read the photo of the card for a long time, so that the step S121 prevents the same user's face and each partial image data of the face printed on the card from being transmitted again, thereby suppressing an excessive number of processing requests (or the number of communications or the amount of communications) (second suppression).
[0175] Furthermore, if the image processing unit 34 does not detect two objects (faces in this case) by the process of step S204 described above (step S212; NO), the detection count is incremented by one (step S223) and the process proceeds to step S224. If the detection count does not exceed a predetermined value (for example, 10 times) (step S224; NO), the process returns to the process of step S221 after waiting for a predetermined time (set cycle value) (step S225). That is, if the first image data and the second image data are transmitted by the transmitting / receiving unit 51 and the object detection unit 35 detects either the first object or the second object again after a state occurs in which the first object and the second object are not detected, the transmitting / receiving unit 51 does not transmit the first image data and the second image data. In this case, the user may turn the photo on the card towards himself / herself for a short period of time on the image acquisition terminal 2, and it is therefore possible that the same person is still within the matching range Q. Therefore, step S121 prevents the partial image data of the same user's face and the face printed on the card from being sent again, thereby achieving the effect of suppressing an excessive number of processing requests (or number of communications or amount of communications) (third suppression).
[0176] Second embodiment Next, the second embodiment will be described with reference to Fig. 24, Fig. 26 and Fig. 30. The second embodiment will be described while explaining the differences from the previous embodiments.
[0177] (Problem of the second embodiment) In the above embodiments, a case has been described in which the distributed data processing terminal 6 requests the centralized data processing server 7 to match two partial image data (matching data and matched data). However, a modified example will be shown in which the centralized data processing server 7 stores feature parameters of the matching data in advance in the storage unit 7000 and one partial image data (matched data) is transmitted from the distributed data processing terminal 6 to the centralized data processing server 7, thereby making matching even easier.
[0178] [Configuration of communication system] Only the differences from the configurations of the communication systems shown in the previous embodiments will be described, and the same points as those in the previous embodiments will be used.
[0179] <Centralized data processing server> The storage unit 7000 of the centralized data processing server is provided with a matching data management table, for example, as shown in Fig. 12. Specifically, this matching data management table is set in advance with items such as the item of the image file to be matched, the feature amount parameters indicating the feature amount of the image file, and the name of the image, and stores the image file, the feature amount parameters, and the name indicating the content of the image file.
[0180] [Processing or Operation of the Present Embodiment] Fig. 26 is a flowchart showing the process of event generation in real-time processing. Fig. 30 is a sequence diagram showing another example of the collation process shown in Fig. 28. Here, differences from Fig. 26 in the second embodiment will be described. Note that there are no major changes to Fig. 26, and therefore a flowchart including changes from Fig. 26 will be omitted.
[0181] First, in the second embodiment, only one partial image data (matching data) is required, so step S212 in Fig. 26 is changed to S212A, and step S221 is changed to S221A, which are judgment steps of "Is there one object (face)?". Similarly, step S214 is changed to step S214A, which are judgment steps of "Is the object (face) within the matching range?".
[0182] Next, the changed processing of FIG. 26 is described as follows.
[0183] First, after the object detection unit 35 sets the number of face detections (count) to 0 (step S211A), if the object detection unit 35 does not detect one object (here, a face) by the process of step S204 described above (step S212A; NO), it waits for a predetermined time (set cycle value) (step S213) and then returns to the process of step S211 again. For example, detection is performed 30 times per second.
[0184] On the other hand, if the object detection unit 35 detects one object (here, a face) by the process of step S204 described above (step S212A; YES), the process proceeds to step S214A. If the detected position is not within the collation possible range Q (step S214A; NO), the process proceeds to step S213.
[0185] Here, the collation possible range Q has already been described with reference to FIG. 29, and therefore will not be described here. If the determination unit 33 determines that the detection position is within the collation possible range Q (step S214A; YES), the object detection unit 35 increments the detection count by one (step S215) and then proceeds to processing in step S216. For example, if an object is at a position indicated by the coordinates (x2, y2) in FIG. 29, it is determined that the object is within the collation possible range Q. Then, if the detection position of the object is within the collation possible range Q (step S216; YES), the process proceeds to processing in step S218. Then, if the detection count does not exceed a predetermined value (for example, 10 times) (step S216; NO), the process returns to processing in step S212A after waiting for a predetermined time (step S217).
[0186] In step S216, the process proceeds to the next step S218 only if the number of detections exceeds a predetermined value. This is to prevent partial image data from being sent from the near-terminal data processing device 5 to the distributed data processing terminal 6 in cases where a user has his or her face photographed and the user does not wish to have his or her face authenticated, turns toward the image acquisition terminal 2 for just a moment, or passes in front of the image acquisition terminal 2, thereby suppressing an excessive number of processing requests, number of communications, and amount of communications (first suppression).
[0187] On the other hand, when the judgment unit 33 judges that the number of detections does not exceed the predetermined value (step S216; YES), the image processing unit 34 encodes the partial image data including the object at the detection position in a general-purpose format such as JPEG (step S218). Then, the event generating unit 36 generates an event message notifying the near terminal data processing device 5 that the partial image data will be transmitted (step S219). Specifically, the event generating unit 36 generates an event message such as "Send" and sets (resets) the number of detections to 0 (step S220), and then proceeds to the processing of step S221A. As a result, the processing of step S421 shown in FIG. 30 is executed, and the partial image data of the user's face is transmitted from the near terminal data processing device 5 to the distributed data processing terminal 6.
[0188] Next, if the image processing unit 34 detects an object (here, a face) by the process of step S204 in Fig. 25 (step S221A; YES), after waiting for a predetermined time (a set cycle value) (step S222), the process returns to the process of step S220 again and the number of occurrences is reset to 0. In this case, since it is considered that the user has been having his / her face read for a long time, the process of step S121A prevents partial image data of the same user's face from being sent again, thereby suppressing an excessive number of processing requests, number of communications, and communication volume (second suppression).
[0189] On the other hand, if the image processing unit 34 does not detect an object (here, a face) by the process of step S204 described above (step S212A; NO), the detection count is incremented by one (step S223), and then the process proceeds to step S224. If the detection count does not exceed a predetermined value (for example, 10 times) (step S224; NO), the process returns to the process of step S221A after waiting for a predetermined time (a set cycle value) (step S225). In this case, the user turns his / her head or looks to the side for a short time at the image acquisition terminal 2, and it is considered that the same person is still within the collation possible range Q. Therefore, the partial image data of the face of the same user is prevented from being transmitted again by step S221A, and an excessive number of processing requests, number of communications, and communication volume are suppressed (third suppression).
[0190] On the other hand, if the number of detections exceeds a predetermined value (for example, 10 times) (step S224; YES), the process returns to step S213.
[0191] As a result, the process of step S111 shown in Fig. 24 is completed as in the previous embodiments. As a result, the communication unit 48 transmits the event message generated in step S219 and the partial image data corrected for distortion in step S205 to the communication unit 58 of the near-terminal data processing device 5 (step S115). Note that, if multiple objects (faces in this case) are detected in the process of step S204, multiple partial image data are transmitted together with one event message. Also, when the communication system is used in a store or the like, the real-time data processing terminal 3 may store the event message and the partial image data, and transmit them together to the near-terminal data processing device 5 after the store closes (for example, at night).
[0192] <Matching process> Next, the process of comparing partial image data will be described with reference to Fig. 30. Fig. 30 is a sequence diagram showing another example of the process of comparing shown in Fig. 28.
[0193] In the distributed data processing terminal 6, when the transmitting / receiving unit 61 receives the partial image data as the data to be matched by the process of step S421 described above, the transmitting / receiving unit 61 transmits matching request information indicating a matching request to the centralized data processing server 7 (S511). This matching request information includes the data to be matched. As a result, the transmitting / receiving unit 71 of the centralized data processing server 7 receives the matching request information.
[0194] Next, in the centralized data processing server 7, the feature generating unit 74 converts the matched data received in the above-mentioned step S511 into bitmap data, and generates feature parameters for identifying an individual, such as the contours and inclinations of the eyes and nose of the face image related to the matched data (S512). Then, the storage / reading unit 79 searches the storage unit 7000 to determine whether or not the feature parameters of the registered matched data remain (S513). If it is determined in step S513 that the feature parameters remain (S513; YES), the matching unit 75 compares the feature parameters of both data (matched data and matched data) to calculate the similarity (S514). Thereafter, by returning to the process of step S513, the processes of steps S513 and S514 are repeated until the similarity is calculated for all the feature parameters of the registered matched data. On the other hand, if it is determined in the process of step S513 that no feature parameters of the matched data remain (including the case where there are none at all), the process proceeds to the process of step S515.
[0195] Next, the memory / readout unit 79 further reads out the “name” corresponding to the feature parameter read out from the memory unit 7000, and temporarily stores it in the memory unit 7000 in combination with the “similarity” calculated in the processing of step S514 (S515).
[0196] Next, the storage / reading unit 79 reads out the similarity assigned to the matching data with the maximum similarity from the similarities temporarily stored in the storage unit 7000 (S516). Then, the transmitting / receiving unit 71 transmits response information indicating a response to the matching request in the above-mentioned step S511 to the distributed data processing terminal 6 (S517). This response information includes the similarity read out in step S516. Here, the response information may include the "name" stored in the storage unit 7000 in the above-mentioned step S515.
[0197] Next, the judgment unit 63 of the distributed data processing terminal 6 compares the similarity with a predetermined threshold (for example, 0.8), and judges the similarity as "OK" if it is greater than the threshold, and judges it as "NG" if it is not (step S518). Then, the display control unit 67 displays a matching result message on the real-time captured image on the display 517 of the distributed data processing terminal 6 (step S519). This matching result message displays the matching result "OK" or "NG".
[0198] [Main Effects of the Present Embodiment] As described above, according to this embodiment, since the matched data is transmitted and received, in addition to the effect of the first embodiment, the communication load can be further reduced.
[0199] 〔supplement〕 In the above embodiment, the real-time data processing terminal 3 judges whether or not the detection count exceeds a predetermined value (steps S216, S224), but it may judge whether or not it is equal to or greater than a threshold value.
[0200] In the above embodiment, the object detection unit 35 detects the position of an object in a captured image by image processing, but the present invention is not limited to this. For example, the object detection unit 35 may measure the distance by combining image processing with LiDAR (Light Detection and Ranging) to irradiate a target object with laser light and measure the reflected light with a detector. This allows, for example, even if an image of two people overlapping one in front and one in back captured by the imaging unit, to recognize them as two people rather than one person, thereby preventing erroneous recognition.
[0201] In the above embodiment, captured image data has been described, but the present invention is not limited to this, and may be created image data created by a user without being captured. In this case, the captured image data and the created image data are examples of "image data". Furthermore, the image acquisition terminal 2 is an example of a communication terminal, and image data may be created on the own terminal without being acquired from an external source. Furthermore, the communication terminal may acquire sound data by collecting sound, may acquire temperature data by a temperature sensor, or may acquire humidity data by a humidity sensor.
[0202] Furthermore, each component such as the CPU 301, 501, 701 may be a single component or a plurality of components. Furthermore, there may be a plurality of image acquisition terminals 2, distributed data processing terminals 6, and centralized data processing servers 7. Furthermore, the distributed data processing terminal 6 may be a server.
[0203] Each function in the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU, and a conventional circuit module designed to execute each of the above-described functions. [Explanation of symbols]
[0204] 2. Image acquisition terminal (an example of a communication terminal) 3 Real-time data processing terminal 5 Near-end data processing device 6 Distributed Data Processing Terminals 7. Centralized Data Processing Server 34 Image processing unit (an example of an image processing means) 35 Object detection unit (an example of an object detection means) 38 Connection section (an example of acquisition means) 51 Transmitting / receiving unit (an example of a transmitting means) 61 Transmitter / receiver 62 Reception 67 Display control section 69 Memory / readout section 71 Transmitter / receiver 74 Feature Generation Unit 75 Matching section 200 Intranet 600 Internet 3001 Image sensor information management DB (an example of image sensor information management means) 3002 Cycle Value Management DB 3003 Image acquisition program DB 3004 Synthesis processing program management DB 3005 Distortion correction program management DB 3006 Service program management DB (an example of program management means) 6000 storage section 6001 Matching data management DB 8000 storage section 8001 Session Management DB 8002 Authentication server management DB 9000 storage section 9001 Authentication Management DB [Prior art documents] [Patent documents]
[0205] [Patent Document 1] Special Publication No. 2016-519812
Claims
1. A distributed processing system having an image acquisition terminal that acquires an image and a data processing terminal that controls a process related to matching based on the image, The image acquisition terminal includes: an image acquisition means for acquiring image data relating to an image of at least one object; an object detection means for detecting at least one object from the image data acquired by the image acquisition means; a transmission means for transmitting, when the number of times a first object and a second object are detected from the image data by the object detection means exceeds a predetermined value, a first partial image data based on the first object and a second partial image data based on the second object in the image data to the data processing terminal, The data processing terminal includes: an image receiving means for receiving the first partial image data and the second partial image data transmitted by the transmitting means; a requesting means for requesting a data processing server capable of communicating via a network to perform a matching process based on the first partial image data and the second partial image data received by the image receiving means; a matching result receiving means for receiving a matching result based on the first partial image data and the second partial image data processed by the data processing server; a display control means for displaying the result of the matching received by the matching result receiving means on a display device, The transmitting means is A distributed processing system characterized in that, if the first object and the second object are detected until a predetermined time has elapsed since the transmission means transmitted the first partial image data and the second partial image data, the first partial image data and the second partial image data are not transmitted to the data processing terminal.
2. The transmitting means is The distributed processing system of claim 1, characterized in that after the first partial image data and the second partial image data are transmitted by the transmitting means, if the first object and the second object are detected before the period during which the first object and the second object are not detected by the object detection means exceeds the specified time, the first partial image data and the second partial image data are not transmitted to the data processing terminal.
3. A distributed processing system having an image acquisition terminal that acquires an image and a data processing terminal that controls a process related to matching based on the image, The image acquisition terminal includes: an image acquisition means for acquiring image data relating to an image of at least one object; an object detection means for detecting at least one object from the image data acquired by the image acquisition means; a transmission means for transmitting, when the number of times a first object and a second object are detected from the image data by the object detection means exceeds a predetermined value, a first partial image data based on the first object and a second partial image data based on the second object in the image data to the data processing terminal, The data processing terminal includes: an image receiving means for receiving the first partial image data and the second partial image data transmitted by the transmitting means; a requesting means for requesting a data processing server capable of communicating via a network to perform a matching process based on the first partial image data and the second partial image data received by the image receiving means; a matching result receiving means for receiving a matching result based on the first partial image data and the second partial image data processed by the data processing server; a display control means for displaying the result of the matching received by the matching result receiving means on a display device, The transmitting means is transmitting the image data acquired by the image acquisition means to the data processing terminal; The display control means A distributed processing system characterized in that, together with the image represented by the image data transmitted by the transmitting means, the result of a comparison based on the first partial image data based on the first object detected from the image data and the second partial image data based on the second object detected from the image data are displayed on the display device.
4. A distributed processing system according to any one of claims 1 to 3, and the data processing server.
5. An image acquisition terminal capable of communicating with a data processing terminal that controls processing related to image-based matching via a network, an image acquisition means for acquiring image data relating to an image of at least one object; an object detection means for detecting at least one object from the image data acquired by the image acquisition means; a transmission means for transmitting, when the number of times a first object and a second object are detected from the image data by the object detection means exceeds a predetermined value, a first partial image data based on the first object and a second partial image data based on the second object in the image data to the data processing terminal, The transmitting means is An image acquisition terminal characterized in that, if the first object and the second object are detected until a predetermined time has elapsed since the transmission means transmitted the first partial image data and the second partial image data, the image acquisition terminal does not transmit the first partial image data and the second partial image data to the data processing terminal.
6. The transmission means: The image acquisition terminal of claim 5, characterized in that after the first partial image data and the second partial image data are transmitted by the transmitting means, if the first object and the second object are detected before the period during which the first object and the second object are not detected by the object detection means exceeds the specified time, the first partial image data and the second partial image data are not transmitted to the data processing terminal.
7. 1. A distributed processing method executed by a distributed processing system having an image acquisition terminal that acquires an image and a data processing terminal that controls processing related to matching based on the image, comprising: The image acquisition terminal includes: An image acquisition step of acquiring image data relating to an image of at least one object; an object detection step of detecting at least one object from the image data acquired by the image acquisition step; a transmission step of transmitting, when the number of times a first object and a second object are detected from the image data by the object detection step exceeds a predetermined value, a first partial image data based on the first object and a second partial image data based on the second object in the image data to the data processing terminal, The data processing terminal includes: an image receiving step of receiving the first partial image data and the second partial image data transmitted by the transmitting step; a request step of requesting a data processing server capable of communicating via a network to perform a matching process based on the first partial image data and the second partial image data received by the image receiving step; a matching result receiving step of receiving a matching result based on the first partial image data and the second partial image data processed by the data processing server; a display control step of displaying the result of the matching received by the matching result receiving step on a display device; A distributed processing method, characterized in that if the first object and the second object are detected until a predetermined time has elapsed since the first partial image data and the second partial image data were transmitted by the transmission step, the first partial image data and the second partial image data are not transmitted to the data processing terminal.
Citation Information
Patent Citations
Face picture recording system and method
JP2004265231A
Apparatus and mobile terminal for dealing with visitor, and system therefor
JP2009100334A
Intercom system
JP2011077835A
Information processing system
JP2013003807A
Information terminal device and information processing method
JP2013196616A
