Information processing device, information processing method, and information processing program

Abstract

To assist in estimating the installation relationship between a plurality of cameras taking images of the traffic environment.SOLUTION: For example, an information processing device 200 includes a determination unit 242 that determines whether a first group of moving objects identified from a first image captured by a first camera and a second group of moving objects identified from a second image captured by a second camera are the same group of moving objects, and an output unit 243 that outputs installation relationship information that can identify the installation relationship between the first camera and the second camera based on movement modes of the first and second moving object groups that are determined to be the same group of moving objects.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present application relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] It is known that cameras installed on roads, roadsides, etc. are calibrated. Patent Document 1 discloses that calibration is performed using a measurement vehicle equipped with a GPS receiver, a data transmitter, landmarks, etc. Patent Document 2 discloses that in camera calibration, when the direction of a line existing on the road plane is input in a captured image, road plane parameters are estimated based on the direction and a direction expressed by an arithmetic expression including road plane parameters. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-10036 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-129942 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, a measurement vehicle is required, and an operator is required for calibration. In Patent Document 2, the lane marks on the road must be manually input into the image, which is a labor-intensive task. For this reason, there is a need for a system that can estimate the installation status of cameras that capture images of the traffic environment without requiring manual work or traffic regulations, as compared to conventional systems that use multiple cameras. [Means for solving the problem]

[0005] An information processing device according to one aspect includes a determination unit that determines whether a first group of moving objects identified from a first image captured by a first camera and a second group of moving objects identified from a second image captured by a second camera are the same group of moving objects, and an output unit that outputs installation relationship information that can identify the installation relationship between the first camera and the second camera based on the movement patterns of the first group of moving objects and the second group of moving objects that have been determined to be the same group of moving objects.

[0006] An information processing method according to one aspect includes a computer determining whether a first group of moving objects identified from a first image captured by a first camera and a second group of moving objects identified from a second image captured by a second camera are the same group of moving objects, and outputting installation relationship information capable of identifying the installation relationship between the first camera and the second camera based on the movement patterns of the first group of moving objects and the second group of moving objects determined to be the same group of moving objects.

[0007] An information processing program according to one aspect causes a computer to determine whether a first group of moving objects identified from a first image captured by a first camera and a second group of moving objects identified from a second image captured by a second camera are the same group of moving objects, and to output installation relationship information capable of identifying the installation relationship between the first camera and the second camera based on the movement patterns of the first group of moving objects and the second group of moving objects that have been determined to be the same group of moving objects. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a camera system according to an embodiment. [Figure 2] FIG. 2 is a diagram schematically showing image information captured by the first camera and the second camera shown in FIG. [Figure 3] FIG. 3 is a diagram illustrating an example of the data structure of the moving object group information. [Figure 4] FIG. 4 is a diagram illustrating an example of the installation relationship information output by the information processing device. [Figure 5]FIG. 5 is a diagram illustrating an example of the configuration of the camera illustrated in FIG. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of a learning device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a processing procedure executed by the camera. [Figure 9] FIG. 9 is a flowchart illustrating an example of a processing procedure executed by the information processing device. [Figure 10] FIG. 10 is a diagram illustrating an example of the operation of the camera and the information processing device in the camera system. [Figure 11] FIG. 11 is a diagram illustrating an example of another configuration of the information processing device according to the embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of another processing procedure executed by the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Several embodiments for implementing an information processing device, an information processing method, an information processing program, etc. according to the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to the following description. Furthermore, the components in the following description include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. In the following description, similar components may be assigned the same reference numerals. Furthermore, duplicated descriptions may be omitted.

[0010] (System Overview) FIG. 1 is a diagram illustrating an example of a camera system according to an embodiment. As shown in FIG. 1, the camera system 1 includes a system that monitors a traffic environment 1000 using multiple cameras 100 installed near the traffic environment 1000. In order to reduce the number of cameras 100 to be installed, the camera system 1 installs the multiple cameras 100 so that the imaging areas of the cameras 100 do not overlap. Furthermore, when the camera system 1 is applied to monitoring a road 1100, the cameras 100 are installed so that they can capture images of moving objects on the road 1100, and the captured images often do not capture buildings around the road 1100. The camera system 1 according to this embodiment provides a function that allows the installation relationship of the multiple cameras 100 to be understood.

[0011] The camera system 1 includes multiple cameras 100 and an information processing device 200. The multiple cameras 100 are installed at different positions in the traffic environment 1000 so as to capture images of the traffic environment 1000 from above. The traffic environment 1000 includes, for example, an ordinary road, a highway, a toll road, and the like. The information processing device 200 has a function of acquiring various information from each of the multiple cameras 100 and estimating the installation relationship of the multiple cameras 100 based on the acquired information. The multiple cameras 100 and the information processing device 200 are configured to be able to communicate via a wired, wireless, or any combination of wired and wireless networks. In the example shown in FIG. 1, for simplicity of explanation, the camera system 1 includes two cameras 100 and one information processing device 200. However, the number of cameras 100 and information processing devices 200 is not limited to this. For example, the camera system 1 may be configured with multiple cameras 100 scattered at multiple intersections on the same road 1100.

[0012] The camera 100 is installed so as to be able to capture an image of the traffic environment 1000, including a road 1100 and a moving object 2000 moving on the road 1100. The moving object 2000 moving on the road 1100 includes, for example, a vehicle, a person, etc. that can move on the road 1100. The moving object 2000 includes, for example, a large automobile, a standard automobile, a large special-purpose automobile, a large motorcycle, a standard motorcycle, a small special-purpose automobile, a bicycle, etc., as defined by the Road Traffic Act, but may also include other vehicles, moving objects, etc. Note that a large automobile includes an automobile with a gross vehicle weight of 8000 kg or more, a maximum load capacity of 5000 kg or more, and an occupancy of 11 or more (such as a bus or truck). The camera 100 can capture an image electronically using an image sensor, for example, a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 100 is installed with its imaging direction facing the road plane of the traffic environment 1000. The camera 100 can be installed, for example, on a road 1100, an intersection, a parking lot, an underground passage, etc. The camera 100 may be capable of capturing at least one of video and still images.

[0013] In the example shown in FIG. 1 , road 1100 has lanes along which moving object 2000 can travel in movement direction 1100M. Moving object 2000 includes standard-sized vehicles 2100, large vehicles 2200, and motorcycles 2300. Standard-sized vehicles 2100 include standard-sized automobiles. Large vehicles 2200 include large automobiles, large special-purpose automobiles, etc. Motorcycles 2300 include large motorcycles, standard-sized motorcycles, etc. Moving object 2000 in movement direction 1100M passes position P1 and moves to position P2.

[0014] Camera 100 includes first camera 100A and second camera 100B. First camera 100A is installed so as to be able to capture an image of position P1 on road 1100 and moving object 2000 moving at position P1. Second camera 100B is installed so as to be able to capture an image of position P2 on road 1100 and moving object 2000 moving at position P2. That is, in camera 100, first camera 100A captures an image of upstream position P1 in moving direction 1100M on road 1100, and second camera 100B captures an image of downstream position P2. First camera 100A and second camera 100B are devices capable of capturing an image of the same moving object 2000 on its moving path.

[0015] Fig. 2 is a diagram showing image information captured by first camera 100A and second camera 100B shown in Fig. 1. In the example shown in Fig. 2, image information D10 shows an image. In the following description, the X-axis direction is the direction of movement of moving object 2000 parallel to the road surface of road 1100, the Y-axis direction is the height direction relative to the road surface, and the Z-axis direction is the direction perpendicular to the X-axis direction.

[0016] 2 , the first camera 100A acquires image information D10 capturing an image of a standard-sized vehicle 2100, a large vehicle 2200, and a motorcycle 2300 traveling on a road 1100. In this case, the first camera 100A identifies the standard-sized vehicle 2100, the large vehicle 2200, and the motorcycle 2300 from the image information D10, and identifies a first moving object group 3100 including the standard-sized vehicle 2100, the large vehicle 2200, and the motorcycle 2300. The first camera 100A generates moving object group information D30 indicating the identification result, and transmits the moving object group information D30 to the information processing device 200.

[0017] Note that the moving object group of the present disclosure may include cases where there is a single moving object 2000 or multiple moving objects 2000. Furthermore, the moving object group information D30 of the present disclosure may include information such as the presence or absence of the moving object 2000, its type, color, movement direction, location, relative position, size or score of the moving object 2000, etc. The type of the moving object 2000 in the moving object group information D30 of the present disclosure may include information such as a light vehicle, a standard-sized vehicle, a large vehicle, a police vehicle, an ambulance, a fire engine, a truck, a bus, a motorcycle, a bicycle, etc. The type of the moving object 2000 in the moving object group information D30 of the present disclosure may include information such as the manufacturer of the moving object 2000, the model name of the moving object 2000, or a combination thereof. The type of the moving object 2000 in the moving object group information D30 of the present disclosure may include information such as a hybrid car, an electric car, an autonomous vehicle, a minivan, a compact car, a sedan, an SUV, or a sports car, or a combination thereof. The location of the moving object 2000 in the moving object group information D30 of the present disclosure may include the position of the moving object 2000's driving lane when there are multiple driving lanes, or the driving order of the multiple moving objects. The size of the moving object 2000 in the moving object group information D30 of the present disclosure may include information on the overall length, overall width, overall height, or total engine displacement of the moving object 2000, or a combination thereof. The score of the present disclosure may be a value expressing the ease of use of the moving object group. The score of the present disclosure may be set so that the value of the moving object group increases as the number of moving objects 2000, the number of types, etc. included in the moving object group increases. This allows the information processing device 200 to improve the accuracy of determining whether the moving objects are the same group by not using moving object groups with low scores in the determination.

[0018] For example, the camera 100 may identify information on the type, color, movement direction, arrangement, relative position, or score of the moving object 2000 included in the identified first moving object group 3100, or any combination thereof. The type includes the type of each moving object 2000 included in the moving object group. The color includes color information of each moving object 2000 included in the moving object group. The movement direction is the movement direction of the moving object 2000 included in the moving object group. The arrangement includes the arrangement (running order) of the moving object 2000 in the moving object group. The relative position includes the position from the center of gravity in the moving object group. The score includes the importance, attention, etc. of the moving object group and each moving object 2000 included in the moving object group.

[0019] Camera 100 can estimate the object captured from image information D10 using object estimation model M1 that has been machine-learned by learning device 300. Camera 100 inputs image information D10 to object estimation model M1, and can estimate the output of object estimation model M1 as the captured object.

[0020] 1, the learning device 300 is, for example, a computer, a server device, or the like. The learning device 300 may or may not be included in the configuration of the camera system 1. The learning device 300 acquires image information D10 obtained by capturing an image of a traffic environment 1000 including a moving object 2000, and a plurality of pieces of first teacher information having correct value information of the moving object 2000. The correct value information may include, for example, information on the image, type, color, or moving direction (orientation) of the moving object 2000, or any combination thereof.

[0021] The learning device 300 generates an object estimation model M1 that estimates the moving object 2000 indicated by input image information D10 through machine learning using multiple pieces of first teacher information. Supervised machine learning can use algorithms such as neural networks, linear regression, and logistic regression. The object estimation model M1 is a model that is trained by machine learning using images of multiple pieces of teacher information and correct answer information to estimate the moving object 2000 indicated by the input image information D10. When image information D10 is input, the object estimation model M1 outputs estimation results that estimate the type, color, movement direction, etc. of the moving object 2000 indicated by the image information D10. The learning device 300 provides the generated object estimation model M1 to the camera 100, thereby enabling the camera 100 to recognize objects.

[0022] The learning device 300 generates a determination model M2 that determines whether or not input multiple pieces of moving object group information D30 belong to the same moving object group through machine learning using multiple pieces of teacher information. The determination model M2 is a model that is machine-learned from multiple pieces of moving object group information D30 as teacher information and correct answer information so as to determine whether or not input multiple pieces of moving object group information D30 belong to the same moving object group. When multiple pieces of moving object group information D30 are input, the determination model M2 outputs a determination result as to whether or not the multiple moving object groups belong to the same moving object group. The learning device 300 provides the generated determination model M2 to the information processing device 200, thereby enabling determination based on the output of the determination model M2. An example of the learning device 300 will be described later.

[0023] FIG. 3 is a diagram showing an example of the data structure of moving object group information D30. As shown in FIG. 3, the moving object group information D30 includes various information such as type, color, movement direction, location, relative position, score, etc. The type of the moving object group information D30 includes the size, such as large or small, large, medium, or small, of the moving objects 2000 of the same type. In this embodiment, a case will be described in which the camera 100 generates the moving object group information D30 based on the identification result and transmits it to the information processing device 200, but this is not limiting. For example, the camera system 1 may have the information processing device 200 implement the function of generating the moving object group information D30.

[0024] As shown in FIG. 2, when the first camera 100A captures image information D10 shown in scene C1, it uses a recognition program, machine learning, or the like to identify the type, color, and movement direction of a standard-sized vehicle 2100, a large vehicle 2200, and a motorcycle 2300 from the image information D10. The first camera 100A then identifies a first moving object group 3100 including the identified multiple moving objects 2000, and identifies a first moving direction 3100M of the first moving object group 3100, as well as the arrangement, relative position, and score of the moving object 2000 within the first moving object group 3100. The first moving direction 3100M is represented by a vector from an origin 3000S of the first moving object group 3100. The first camera 100A generates moving object group information D30 for the first camera 100A based on the identification result. The first camera 100A transmits the generated moving object group information D30 to the information processing device 200.

[0025] For example, the first camera 100A can capture an image of the traffic environment 1000 including the first moving object group 3100 and a moving object 2400 that can move in a direction intersecting with the first movement direction 3100M of the first moving object group 3100. In this case, the first camera 100A can identify the moving object group of the moving object 2400 that can move in a direction intersecting with the first movement direction 3100M of the first moving object group 3100. In other words, the first camera 100A can identify moving object groups moving in different directions. In the following description, when it is not necessary to distinguish between the first moving object group 3100 and the second moving object group 3200, they will be simply referred to as the moving object group 3000.

[0026] After that, the standard-sized vehicle 2100, the large vehicle 2200, and the motorcycle 2300 pass position P1 on the road 1100 and move to position P2 on the road 1100. When the standard-sized vehicle 2100, the large vehicle 2200, and the motorcycle 2300 reach position P2 on the road 1100, the second camera 100B captures image information D10 shown in scene C2. The second camera 100B identifies the type, color, movement direction, etc. of the standard-sized vehicle 2100, the large vehicle 2200, and the motorcycle 2300 from the image information D10 using a recognition program, machine learning, etc. The second camera 100B identifies a second moving object group 3200 including the identified multiple moving objects 2000, and identifies a second movement direction 3200M of the second moving object group 3200, as well as the arrangement, relative position, and score of the moving object 2000 in the second moving object group 3200. Second movement direction 3200M is represented by a vector from origin 3000S of second moving object group 3200. Second camera 100B generates moving object group information D30 of second camera 100B based on the identification result. Second camera 100B transmits the generated moving object group information D30 to information processing device 200.

[0027] Generally, when multiple moving objects 2000 move along a road 1100, even if they do not strictly match, their locations and travel order generally do not change. Focusing on this, upon receiving moving object group information D30 from the first camera 100A and the second camera 100B, the information processing device 200 determines whether the first moving object group 3100 captured by the first camera 100A and the second moving object group 3200 captured by the second camera 100B are the same moving object group 3000. The information processing device 200 can determine that the moving objects are the same moving object group if the degree of similarity of the information on the type, color, location, or position of the multiple moving objects 2000, or any combination thereof, indicated by the two pieces of moving object group information D30, exceeds a determination threshold. For example, the information processing device can determine whether the moving objects are the same moving object group 3000 using a determination model that a learning device has machine-learned from the multiple pieces of moving object group information D30 and correct answer information.

[0028] The information processing device 200 outputs installation relationship information capable of identifying the installation relationship between the first camera 100A and the second camera 100B based on the movement patterns of the first moving object group 3100 and the second moving object group 3200 determined to belong to the same moving object group 3000. The information processing device 200, for example, estimates the interrelationship between the multiple cameras 100 and outputs installation relationship information capable of identifying the relative angles, relative positions, etc. of the multiple cameras 100. In this embodiment, the information processing device 200 estimates the interrelationship between the multiple cameras 100 by determining how the moving objects 2000 captured by the first camera 100A and the second camera 100B appear as a group between the multiple cameras 100. The movement patterns in the present disclosure may include the speed, direction, position, driving lane, relative distance from other moving objects, relative speed from other moving objects, relative position from other moving objects, or any combination thereof.

[0029] FIG. 4 is a diagram showing an example of installation relationship information D100 output by the information processing device 200. As shown in FIG. 4, the information processing device 200 outputs the installation relationship information D100 generated based on information such as image information D10 and a traffic environment 1000. In this embodiment, the installation relationship information D100 is information that enables identification of estimated relative angles and relative positions. The installation relationship information D100 has an origin D101, an X-axis D102, and a Z-axis D103 for multiple cameras 100, based on the same movement direction 1100M of the same road 1100. The origin D101 indicates the axis of the road surface of the road 1100 for each camera 100. The X-axis D102 indicates the X-axis direction relative to the origin D101. The Z-axis D103 indicates the Z-axis direction relative to the origin D101 and is perpendicular to the X-axis D102.

[0030] In the example shown in FIG. 4, the installation relationship information D100 has an origin D101, an X-axis D102, and a Z-axis D103 corresponding to the first camera 100A and the second camera 100B. The installation relationship information D100 indicates the relative angle of the origin D101 of the second camera 100B with respect to the origin D101 of the first camera 100A, where the origin D101 of the second camera 100B is rotated 130 degrees. The installation relationship information D100 indicates the relative position of the origin D101 of the second camera 100B with respect to the origin D101 of the first camera 100A, where the origin D101 of the second camera 100B is 45 m away in the X-axis direction and 8 m away in the Z-axis direction. As a result, the information processing device 200 can output the installation relationship information D100 to allow the interrelationships between the multiple installed cameras 100 to be recognized. In this embodiment, the installation relationship information D100 indicates the mutual relationship regarding the installation of two cameras 100, but may be information indicating the mutual relationship between cameras 100 in two or more locations. The installation relationship information D100 of the present disclosure may include the distance between the multiple cameras 100, the relative imaging directions of the multiple cameras 100, the installation heights of the multiple cameras 100 from the ground, or any combination thereof.

[0031] (Example of camera configuration) Fig. 5 is a diagram showing an example of the configuration of camera 100 shown in Fig. 1. As shown in Fig. 5, each of the multiple cameras 100 includes an imaging unit 110, a sensor unit 120, a communication unit 130, a storage unit 140, and a control unit 150. The control unit 150 is electrically connected to the imaging unit 110, the sensor unit 120, the communication unit 130, the storage unit 140, etc.

[0032] In this embodiment, the camera 100 is described as having an imaging unit 110, a sensor unit 120, a communication unit 130, a storage unit 140, and a control unit 150, but is not limited to this. For example, the camera 100 may be configured to have the imaging unit 110, the communication unit 130, the storage unit 140, and the control unit 150.

[0033] The imaging unit 110 is installed so as to be able to capture an image of a traffic environment 1000 including a road 1100 and a moving object 2000 moving on the road 1100. The imaging unit 110 can electronically capture image information D10 using an image sensor such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 110 can capture images of the traffic environment 1000 in real time at a predetermined frame rate and supply the captured image information D10 to the control unit 150.

[0034] The sensor unit 120 detects sensor information that can identify the state of the camera 100. The sensor unit 120 can use various sensors, such as a position sensor and a gyro sensor. An example of a position sensor is a sensor that acquires absolute coordinate positions, such as a GPS (Global Positioning System) receiver. The sensor unit 120 can supply sensor information including the installation position and installation angle of the camera 100 to the control unit 150. This allows the control unit 150 to acquire information such as the self-position and installation state of the camera 100 based on the sensor information.

[0035] The communication unit 130 can communicate with, for example, the information processing device 200, other communication devices, etc. The communication unit 130 can support various communication standards. The communication unit 130 can send and receive various types of data via, for example, a wired or wireless network, etc. The communication unit 130 can supply the received data to the control unit 150. The communication unit 130 can send data to a destination instructed by the control unit 150.

[0036] The storage unit 140 can store programs and various information (data). The storage unit 140 is also used as a working area for temporarily storing processing results of the control unit 150. The storage unit 140 may include any non-transitory storage medium, such as a semiconductor storage medium or a magnetic storage medium. The storage unit 140 may include multiple types of storage media. The storage unit 140 may include a combination of a portable storage medium, such as a memory card, an optical disk, or a magneto-optical disk, and a storage medium reader. The storage unit 140 may include a storage device used as a temporary storage area, such as a RAM (Random Access Memory).

[0037] The storage unit 140 can store various types of information, such as a program 141, image information D10, moving object group information D30, installation state information D40, an object estimation model M1, and a state estimation model M10. The program 141 is a program that causes the control unit 150 to execute functions for realizing processes related to various operations of the camera 100. The image information D10 includes information indicating an image captured by the imaging unit 110. The moving object group information D30 includes information indicating a moving object group 3000 identified from the image information D10 captured by the imaging unit 110. The installation state information D40 includes information that can identify the installation state, installation position, etc. of the camera 100. The object estimation model M1 is a machine learning model provided by the learning device 300, and is used to estimate the moving object 2000 in the input image information D10. The state estimation model M10 is a machine learning model trained to estimate installation state parameters of the camera 100 that captured the input image information D10. The state estimation model M10 is used to estimate installation state parameters of the camera 100 that captured the image information D10.

[0038] The control unit 150 is an arithmetic processing device. Examples of the arithmetic processing device include, but are not limited to, a central processing unit (CPU), a system-on-a-chip (SoC), a micro control unit (MCU), a field-programmable gate array (FPGA), and a coprocessor. The control unit 150 can comprehensively control the operation of the camera 100 to realize various functions.

[0039] Specifically, the control unit 150 can execute instructions included in the program 141 stored in the storage unit 140 while referring to information stored in the storage unit 140 as necessary. The control unit 150 then controls the functional units in accordance with the data and instructions, thereby realizing various functions. The functional units include, but are not limited to, the sensor unit 120 and the communication unit 130, for example.

[0040] The control unit 150 has functional units such as an image recognition unit 151, a classification unit 152, an estimation unit 153, and a transmission unit 154. The control unit 150 realizes the functional units such as the image recognition unit 151, the classification unit 152, the estimation unit 153, and the transmission unit 154 by executing the program 141. The program 141 is a program for causing the control unit 150 of the camera 100 to function as the image recognition unit 151, the classification unit 152, the estimation unit 153, and the transmission unit 154.

[0041] The image recognition unit 151 recognizes the moving object 2000 from the image indicated by the image information D10 captured by the imaging unit 110. The image recognition unit 151 has a function of recognizing the presence, type, color, moving direction, arrangement, relative position, score, etc. of the moving object 2000 in the image using, for example, a machine learning model that has been trained to recognize the moving object 2000, an image recognition program, etc.

[0042] The identification unit 152 identifies a moving object group 3000 including at least one moving object 2000 from the image captured by the imaging unit 110 based on the recognition result of the image recognition unit 151. The identification unit 152 identifies the moving object group 3000 based on the presence or absence, type, color, movement direction, arrangement, relative position, score, etc. of the moving object 2000 in the image, and identifies the movement direction, origin (center) of the moving object group 3000, etc. The identification unit 152 generates moving object group information D30 capable of identifying the identified moving object group 3000, and stores the information in the storage unit 140.

[0043] The estimation unit 153 can provide a function of estimating installation state parameters of the camera 100 that captured input image information D10 using the state estimation model M10 generated by the learning device 300. The installation state parameters include, for example, the installation angle and installation position of the camera 100. The installation state parameters may also include, for example, the number of pixels of the camera 100 and the size of the image. The estimation unit 153 inputs the image information D10 to the state estimation model M10 and can estimate the installation state parameters of the camera 100 based on the output of the state estimation model M10. The estimation unit 153 generates installation state information D40 indicating the installation angle, installation position, etc. of the camera 100 based on the estimated installation state parameters and stores the information in the storage unit 140.

[0044] The transmission unit 154 can provide a function of transmitting various types of information to the information processing device 200 via the communication unit 130. The transmission unit 154 transmits image information D10, moving object group information D30, installation state information D40, and the like to the information processing device 200 via the communication unit 130. The transmission unit 154 controls the communication unit 130 so as to transmit the moving object group information D30 and the installation state information D40 to the information processing device 200 when the camera 100 is installed, when maintenance is performed, and the like.

[0045] An example of the functional configuration of the camera 100 according to this embodiment has been described above. Note that the configuration described above using Fig. 5 is merely an example, and the functional configuration of the camera 100 according to this embodiment is not limited to this example. The functional configuration of the camera 100 according to this embodiment can be flexibly modified according to specifications and operation.

[0046] (Example of learning device configuration) Fig. 6 is a diagram illustrating an example of the configuration of a learning device 300 according to an embodiment. As shown in Fig. 6, the learning device 300 includes a display unit 310, an operation unit 320, a communication unit 330, a storage unit 340, and a control unit 350. The control unit 350 is electrically connected to the display unit 310, the operation unit 320, the communication unit 330, the storage unit 340, and the like. In this embodiment, an example will be described in which the learning device 300 performs machine learning using a convolutional neural network (CNN), which is a type of neural network. As is well known, a CNN has an input layer, an intermediate layer, and an output layer.

[0047] Display unit 310 is configured to be able to display various types of information under the control of control unit 350. Display unit 310 has a display panel such as a liquid crystal display, an organic EL display, etc. Display unit 310 displays information such as characters, figures, and images in response to signals input from control unit 350.

[0048] The operation unit 320 has one or more devices for accepting user operations. The devices for accepting user operations include, for example, keys, buttons, a touch screen, a mouse, etc. The operation unit 320 can supply a signal corresponding to the accepted operation to the control unit 350.

[0049] The communication unit 330 can communicate with, for example, the camera 100, the information processing device 200, other communication devices, etc. The communication unit 330 can support various communication standards. The communication unit 330 can send and receive various types of data via, for example, a wired or wireless network. The communication unit 330 can supply the received data to the control unit 350. The communication unit 330 can send data to a destination instructed by the control unit 350.

[0050] The storage unit 340 can store programs and various information. The storage unit 340 is also used as a working area for temporarily storing processing results of the control unit 350. The storage unit 340 may include any non-transitory storage medium, such as a semiconductor storage medium or a magnetic storage medium. The storage unit 340 may include multiple types of storage media. The storage unit 340 may include a combination of a portable storage medium, such as a memory card, an optical disk, or a magneto-optical disk, and a storage medium reader. The storage unit 340 may include a storage device, such as RAM, that is used as a temporary storage area.

[0051] The storage unit 340 can store various types of information, such as a program 341, training information 342, an object estimation model M1, a determination model M2, a state estimation model M10, and an installation relationship estimation model M20. The program 341 causes the control unit 350 to execute a function of using CNN to generate an object estimation model M1 that estimates an object indicated by image information D10. The program 341 causes the control unit 350 to execute a function of using CNN to generate a determination model M2 that determines whether multiple pieces of moving object group information D30 belong to the same moving object group 3000. The program 341 causes the control unit 350 to execute a function of using CNN to generate a state estimation model M10 that estimates installation state parameters of the camera 100 that captured the image information D10. The program 341 causes the control unit 350 to execute a function of using CNN to generate an installation relationship estimation model M20 that estimates the installation relationship of the multiple cameras 100.

[0052] The teacher information 342 includes learning information, training information, etc. used in machine learning. The teacher information 342 has information that combines image information D10 used in machine learning for object estimation and correct value information D21 associated with the image information D10. The image information D10 is input information for supervised learning. For example, the image information D10 indicates a color image captured of the traffic environment 1000 including the moving object 2000. The correct value information D21 is correct answer information for supervised machine learning. The correct value information D21 includes, for example, information such as the image, type, color, and moving direction (orientation) of the moving object 2000.

[0053] The teacher information 342 includes information that combines moving object group information D30 used for machine learning to determine the moving object group 3000 and correct answer value information D22 for machine learning to determine whether the moving object group information D30 is the same moving object group 3000. For example, the correct answer value information D22 includes multiple pieces of moving object group information D30 that are determined to be the same moving object group 3000. The correct answer value information D22 includes information such as the image, arrangement, and movement direction of the moving object group 3000 for determining that they are the same moving object group 3000.

[0054] The teacher information 342 includes data combining image information D10 used in machine learning of state estimation and correct value information D23 associated with the image information D10. The image information D10 is input information for supervised learning. For example, the image information D10 indicates a color image of a traffic environment 1000 including a moving object 2000, and has a pixel count of 1280 x 960. The correct value information D23 includes information indicating installation state parameters of the camera 100 that captured the image information D10. The correct value information D23 is correct answer information for supervised machine learning. The correct value information D23 includes data indicating six parameters (values), for example, the installation angle (α, β, γ) and installation position (x, y, z) of the camera 100.

[0055] The teacher information 342 includes information that combines the moving object group information D30 and the installation state information D40 used in machine learning to estimate the installation relationship of the multiple cameras 100, and the correct value information D24 for estimating the installation relationship from the moving object group information D30 and the installation state information D40. For example, the correct value information D24 includes information for estimating the installation relationship information D100 based on the relationship between the movement directions of the multiple moving object group information D30 and the installation states, etc., indicated by the installation state information D40 of the multiple cameras 100.

[0056] The object estimation model M1 is a learning model generated by extracting the features, regularities, patterns, etc. of the moving body 2000 indicated by the image information D10 using the image information D10 and correct value information D21 contained in the teacher information 342, and by machine learning the relationship with the correct value information D21. When image information D10 is input, the object estimation model M1 predicts the teacher information 342 that is similar to the features, etc. of the moving body 2000 indicated by the image information D10, estimates the position, orientation, speed, and vehicle type of the object indicated by the image information D10, and outputs the estimation results.

[0057] The determination model M2 is a learning model generated by machine learning to extract features, regularities, patterns, etc. of the moving object group 3000 indicated by the moving object group information D30 using the plurality of pieces of moving object group information D30 and the correct value information D22 contained in the teacher information 342, and to determine whether the plurality of moving object groups 3000 are the same moving object group 3000. When the plurality of pieces of moving object group information D30 are input, the determination model M2 determines whether the plurality of moving object groups 3000 indicated by the plurality of pieces of moving object group information D30 are the same moving object group 3000, and outputs the determination result. The determination model M2 can determine whether the plurality of moving object groups 3000 indicated by the plurality of pieces of moving object group information D30 are the same moving object group 3000, based on at least one of type information, location information, and color information of the moving objects 2000 of the moving object group 3000.

[0058] The determination model M2 can determine type information of the moving bodies 2000 in the first moving body group 3100 based on a first size of each moving body 2000 relative to the size of the first moving body group 3100, and can determine type information of the moving bodies 2000 in the second moving body group based on a second size of each moving body 2000 relative to the size of the second moving body group 3200. The determination model M2 can determine whether the first moving body group 3100 and the second moving body group 3200 are the same moving body group based on a first relative position of each moving body 2000 with the center of gravity of the first moving body group 3100 as the origin and a second relative position of each moving body 2000 with the center of gravity of the second moving body group 3200 as the origin. The determination model M2 can determine whether the first moving object group 3100 and the second moving object group 3200 are the same moving object group when the scores of the first moving object group 3100 and the second moving object group 3200 satisfy a determination condition.

[0059] The state estimation model M10 is a learning model generated by extracting features, regularities, patterns, etc. of the image information D10 using the image information D10 and correct value information D23 contained in the teacher information 342, and by machine learning the relationship with the correct value information D23. When image information D10 is input, the state estimation model M10 predicts teacher information 342 that is similar to the features, etc. of the image information D10, estimates installation state parameters of the camera 100 that captured the image information D10 based on the correct value information D23, and outputs the estimation results.

[0060] The installation relationship estimation model M20 is a learning model generated by machine learning the installation relationship of the multiple cameras 100 based on how the moving object group 3000 appears (image capture state) to the multiple cameras 100, using the moving object group information D30, installation state information D40, and correct value information D24 contained in the teacher information 342. When the multiple moving object group information D30 and installation state information D40 are input, the installation relationship estimation model M20 predicts teacher information 342 that is similar to how the moving object group 3000 appears to the multiple cameras 100, estimates the installation relationship of the multiple cameras 100 based on the correct value information D24, and outputs the estimation result.

[0061] The control unit 350 is a processing unit. Examples of the processing unit include, but are not limited to, a CPU, an SoC, an MCU, an FPGA, and a coprocessor. The control unit 350 can comprehensively control the operation of the learning device 300 to realize various functions.

[0062] Specifically, the control unit 350 can execute instructions included in the program 341 stored in the storage unit 340 while referring to information stored in the storage unit 340 as necessary. The control unit 350 then controls the functional units in accordance with the data and instructions, thereby realizing various functions. The functional units include, but are not limited to, the display unit 310 and the communication unit 330, for example.

[0063] The control unit 350 has functional units such as a first acquisition unit 351, a first machine learning unit 352, a second acquisition unit 353, a second machine learning unit 354, a third acquisition unit 355, a third machine learning unit 356, a fourth acquisition unit 357, and a fourth machine learning unit 358. The control unit 350 executes a program 341 to realize the functions of the first acquisition unit 351, the first machine learning unit 352, the second acquisition unit 353, the second machine learning unit 354, the third acquisition unit 355, the third machine learning unit 356, the fourth acquisition unit 357, and the fourth machine learning unit 358. The program 341 is a program for causing the control unit 350 of the learning device 300 to function as the first acquisition unit 351, the first machine learning unit 352, the second acquisition unit 353, the second machine learning unit 354, the third acquisition unit 355, the third machine learning unit 356, the fourth acquisition unit 357, and the fourth machine learning unit 358.

[0064] The first acquisition unit 351 acquires image information D10 obtained by capturing an image of the traffic environment 1000 including the moving object 2000 and correct answer value information D21 as teacher information 342. The first acquisition unit 351 acquires the image information D10 and the correct answer value information D21 from a preset storage destination, a storage destination selected by the operation unit 320, or the like, and stores the image information D10 and the correct answer value information D21 in association with the teacher information 342 in the memory unit 340. The first acquisition unit 351 acquires a plurality of pieces of image information D10 and correct answer value information D21 to be used for machine learning.

[0065] The first machine learning unit 352 generates an object estimation model M1 that estimates the moving body 2000 indicated by the input image information D10 through machine learning using the plurality of pieces of teacher information 342 acquired by the first acquisition unit 351. The first machine learning unit 352 constructs a CNN based on, for example, the teacher information 342. The CNN receives the image information D10 as input, and a network is constructed to output an estimation result for the image information D10. The estimation result includes information such as the position, orientation, speed, and vehicle type of the moving body 2000 (object) indicated by the image information D10.

[0066] The second acquisition unit 353 acquires a plurality of pieces of moving object group information D30 and correct value information D22 as teacher information 342. The second acquisition unit 353 acquires the moving object group information D30 and the correct value information D22 from a preset storage destination, a storage destination selected by the operation unit 320, or the like, and stores the information in association with the teacher information 342 in the storage unit 340. The second acquisition unit 353 acquires a plurality of pieces of moving object group information D30 and correct value information D22 to be used for machine learning.

[0067] The second machine learning unit 354 generates a determination model M2 that determines whether or not the multiple moving object groups 3000 indicated by the input multiple pieces of moving object group information D30 are the same moving object group 3000, through machine learning using the multiple pieces of teacher information 342 acquired by the second acquisition unit 353. The second machine learning unit 354 constructs a CNN based on, for example, the teacher information 342. The CNN receives the multiple pieces of moving object group information D30 as input, and constructs a network so as to output a determination result as to whether or not the multiple moving object groups 3000 indicated by the input multiple pieces of moving object group information D30 are the same moving object group 3000.

[0068] For example, the determination model M2 is machine-learned to compare the movement directions of the multiple moving object groups 3000 and determine, based on the comparison results and a correct answer value, whether the multiple moving object groups 3000 are the same moving object group 3000. For example, the determination model M2 is machine-learned to compare combinations of type information of the multiple moving objects 2000 included in the moving object group 3000 and determine, based on the comparison results and a correct answer value, whether the multiple moving object groups 3000 are the same moving object group 3000. For example, the determination model M2 is machine-learned to compare combinations of type, color, and location of the multiple moving objects 2000 included in the moving object group 3000 and the relative positions of the moving object groups 3000 centered on the origin 3000S, and determine, based on the comparison results and a correct answer value, whether the multiple moving object groups 3000 are the same moving object group 3000.

[0069] The third acquisition unit 355 acquires image information D10 of an image of the traffic environment 1000 including the moving object 2000 and correct value information D23 of the installation state parameters of the camera 100 that captured the image information D10 as teacher information 342. The third acquisition unit 355 acquires the image information D10 and the correct value information D23 from a preset storage destination, a storage destination selected by the operation unit 320, or the like, and stores the image information D10 and the correct value information D23 in association with the teacher information 342 in the memory unit 340. The third acquisition unit 355 acquires a plurality of pieces of image information D10 and correct value information D23 to be used for machine learning.

[0070] The third machine learning unit 356 generates a state estimation model M10 that estimates installation state parameters of the camera 100 that captured the input image information D10, through machine learning using the plurality of pieces of teacher information 342 acquired by the third acquisition unit 355. The third machine learning unit 356 constructs a CNN based on, for example, the teacher information 342. The CNN receives the image information D10 as input, and a network is constructed to output a classification result for the image information D10. The classification result includes information for estimating installation state parameters of the camera 100 that captured the image information D10.

[0071] The fourth acquisition unit 357 acquires, as teacher information 342, moving object group information D30, installation state information D40 associated with the moving object group information D30, and correct value information D24 indicating the installation relationship of the multiple cameras 100 corresponding to that information. The fourth acquisition unit 357 acquires the moving object group information D30, the installation state information D40, and the correct value information D24 from a preset storage destination, a storage destination selected by the operation unit 320, or the like, and stores them in association with the teacher information 342 in the storage unit 340. The fourth acquisition unit 357 acquires multiple pieces of moving object group information D30, installation state information D40, and correct value information D24 to be used for machine learning.

[0072] The fourth machine learning unit 358 generates an installation relationship estimation model M20 that estimates the installation relationship of the multiple cameras 100 corresponding to the input moving object group information D30 and installation state information D40, through machine learning using the multiple pieces of teacher information 342 acquired by the fourth acquisition unit 357. The fourth machine learning unit 358 constructs a CNN based on, for example, the teacher information 342. The CNN receives the multiple pieces of moving object group information D30 and installation state information D40 as input, and constructs a network to estimate the installation relationship of the multiple cameras 100. In other words, the fourth machine learning unit 358 generates an installation relationship estimation model M20 that estimates the installation relationship of the multiple cameras based on how the moving object group 3000 appears in the multiple cameras 100.

[0073] An example of the functional configuration of the learning device 300 according to this embodiment has been described above. Note that the configuration described above using Fig. 6 is merely an example, and the functional configuration of the learning device 300 according to this embodiment is not limited to this example. The functional configuration of the learning device 300 according to this embodiment can be flexibly modified according to specifications and operations.

[0074] In this embodiment, the learning device 300 generates the object estimation model M1, the determination model M2, the state estimation model M10, and the installation relationship estimation model M20, but is not limited to this. The learning device 300 may be realized by a plurality of devices that individually generate the object estimation model M1, the determination model M2, the state estimation model M10, and the installation relationship estimation model M20.

[0075] (Configuration example of information processing device) Fig. 7 is a diagram showing an example of the configuration of an information processing device 200 according to an embodiment. As shown in Fig. 7, the information processing device 200 includes an input unit 210, a communication unit 220, a storage unit 230, and a control unit 240. The control unit 240 is electrically connected to the input unit 210, the communication unit 220, the storage unit 230, etc. In the following description, the information processing device 200 will be described as outputting results on an external electronic device, but the information processing device 200 may also include an output device such as a display device.

[0076] Image information D10 captured by camera 100 is input to input unit 210. Input unit 210 has a connector that can be electrically connected to camera 100 via a cable, for example. Input unit 210 supplies image information D10 input from camera 100 to control unit 240.

[0077] The communication unit 220 can communicate with, for example, the camera 100, the learning device 300, a management device that manages the camera 100, etc. The communication unit 220 can support various communication standards. The communication unit 220 can send and receive various information via, for example, a wired or wireless network. The communication unit 220 can supply the received information to the control unit 240. The communication unit 220 can send information to a destination instructed by the control unit 240.

[0078] The storage unit 230 can store programs and information. The storage unit 230 is also used as a working area for temporarily storing processing results of the control unit 240. The storage unit 230 may include any non-transitory storage medium, such as a semiconductor storage medium or a magnetic storage medium. The storage unit 230 may include multiple types of storage media. The storage unit 230 may include a combination of a portable storage medium, such as a memory card, an optical disk, or a magneto-optical disk, and a storage medium reader. The storage unit 230 may include a storage device, such as RAM, that is used as a temporary storage area.

[0079] The storage unit 230 can store, for example, a program 231, setting information 232, image information D10, moving object group information D30, installation state information D40, installation relationship information D100, a determination model M2, an installation relationship estimation model M20, etc. The program 231 causes the control unit 240 to execute various control functions for operating the information processing device 200. The setting information 232 includes various information such as various settings related to the operation of the information processing device 200 and settings related to the installation state of the cameras 100 to be managed. The storage unit 230 can store multiple pieces of image information D10 and moving object group information D30 in chronological order while associating them with each other. The storage unit 230 can store installation state information D40 and installation relationship information D100 corresponding to each of the multiple cameras 100 while associating them with each other. The determination model M2 is a machine learning model generated by the learning device 300. The installation relationship estimation model M20 is a machine learning model generated by the learning device 300.

[0080] The control unit 240 is an arithmetic processing device. Examples of the arithmetic processing device include, but are not limited to, a CPU, an SoC, an MCU, an FPGA, and a coprocessor. The control unit 240 comprehensively controls the operation of the information processing device 200 to realize various functions.

[0081] Specifically, the control unit 240 executes instructions contained in the program 231 stored in the storage unit 230 while referring to the information stored in the storage unit 230 as necessary. The control unit 240 then controls the functional units in accordance with the information and instructions, thereby realizing various functions. The functional units include, but are not limited to, the input unit 210 and the communication unit 220, for example.

[0082] The control unit 240 has functional units such as an acquisition unit 241, a determination unit 242, and an output unit 243. The control unit 240 realizes the functional units such as the acquisition unit 241, the determination unit 242, and the output unit 243 by executing the program 231. The program 231 is a program for causing the control unit 240 of the information processing device 200 to function as the acquisition unit 241, the determination unit 242, and the output unit 243.

[0083] The acquisition unit 241 acquires image information D10 and moving object group information D30 from the multiple cameras 100 via the communication unit 220. For example, the acquisition unit 241 acquires moving object group information D30 from the first camera 100A as first moving object group information, and acquires moving object group information D30 from the second camera 100B as second moving object group information. The acquisition unit 241 associates the moving object group information D30 acquired from the first camera 100A and the moving object group information D30 acquired from the second camera 100B during the same time period on the same day, and stores them in the storage unit 230.

[0084] The determination unit 242 determines whether the first moving object group 3100 identified from the image captured by the first camera 100A and the second moving object group 3200 identified from the image captured by the second camera 100B are the same moving object group 3000. The determination unit 242 determines whether the moving objects 2000 are the same moving object group 3000 using information such as the type, color, movement direction, arrangement, relative position, and score of the moving objects 2000 included in the multiple moving object groups 3000. In this embodiment, the determination unit 142 makes the determination using a machine-learned determination model M2. The determination unit 242 inputs multiple moving object group information D30 into the determination model M2 and sets the output of the determination model M2 as the determination result. Note that, when the determination result of the determination model M2 includes reliability, the determination unit 242 may validate the determination of whether the moving objects 3000 are the same moving object group 3000 if the reliability is high, and invalidate the determination of whether the moving objects 3000 are the same moving object group 3000 if the reliability is low.

[0085] The output unit 243 outputs installation relationship information D100 that can identify the installation relationship between the first camera 100A and the second camera 100B based on the movement patterns of the first moving body group 3100 and the second moving body group 3200 that have been determined to belong to the same moving body group 3000. The output of the output unit 243 includes, for example, generating the installation relationship information D100 and outputting the installation relationship information D100 to an external electronic device, output device, or the like. The output unit 243 outputs the installation relationship information D100 that can identify the installation relationship between the first camera 100A and the second camera 100B based on the first movement direction 3100M of the first moving body group 3100 and the second movement direction 3200M of the second moving body group 3200. In this embodiment, the output unit 243 estimates the installation relationship using a machine-learned installation relationship estimation model M20. The output unit 243 inputs a plurality of pieces of moving object group information D30 and installation state information D40 into the installation relationship estimation model M20, and outputs the installation relationship estimation model M20 as the estimation result.

[0086] The output unit 143 outputs statistical installation relationship information D100 based on time-series information of the first moving object group 3100 and time-series information of the second moving object group 3200. For example, the output unit 243 outputs installation relationship information D100 that statistically indicates an estimated result of the installation relationship between the first camera 100A and the second camera 100B for each of a plurality of different combinations of the first moving object group 3100 and the second moving object group 3200. The output unit 243 outputs installation relationship information D100 that enables identification of the mutual relationship between the first camera 100A and the second camera 100B (see, for example, FIG. 4).

[0087] The output unit 243 outputs installation relationship information D100 capable of identifying the scores of the first moving body group 3100 and the second moving body group 3200. The output unit 243 outputs installation relationship information D100 capable of identifying a score that increases as the number of moving bodies 2000 or the number of type information included in the moving body group 3000 increases. The output unit 243 recognizes the number of moving bodies 2000 and the number of type information included in the moving body group 3000 based on the moving body group information D30, and adds information indicating a score corresponding to the recognized number to the installation relationship information D100.

[0088] The output unit 243 terminates the output operation when the number of the moving object group 3000 determined by the determination unit 242 becomes equal to or greater than a specified number. As a result, the information processing device 200 can appropriately terminate the estimation of the installation relationship by using the determined number of the moving object group 3000 as a termination condition.

[0089] An example of the functional configuration of the information processing device 200 according to this embodiment has been described above. Note that the above configuration described using Fig. 7 is merely an example, and the functional configuration of the information processing device 200 according to this embodiment is not limited to this example. The functional configuration of the information processing device 200 according to this embodiment can be flexibly modified according to specifications and operations.

[0090] (Example of camera processing procedure) 8 is a flowchart showing an example of a processing procedure executed by the camera 100. The processing procedure shown in FIG.

[0091] 8, the control unit 150 of the camera 100 estimates the installation state of the camera 100 (step S100). For example, the control unit 150 estimates installation state parameters of the camera 100 that captured input image information D10 using a machine-learned installation relationship estimation model M20 generated by the learning device 300. The control unit 150 generates installation state information D40 based on the installation state parameters estimated by the installation relationship estimation model M20, position information detected by the sensor unit 120, and the like, and stores the information in the storage unit 140. When the process of step S100 ends, the control unit 150 advances the process to step S101.

[0092] The control unit 150 identifies the moving object 2000 in the image information D10 captured by the imaging unit 110 (step S101). For example, the control unit 150 inputs the image information D10 to the object estimation model M1, and identifies the moving object 2000 indicated by the image information D10 based on the output of the object estimation model M1. The identification result includes information such as the type, color, moving direction (orientation), and placement in the image of the moving object 2000. The control unit 150 stores the identification result in the storage unit 140 in association with the image information D10, and then proceeds to step S102.

[0093] The control unit 150 generates moving object group information D30 (step S102). For example, based on the identification results of the multiple moving objects 2000 identified in step S101, the control unit 150 generates moving object group information D30 that can identify information such as the type, color, and movement direction of the moving object 2000, and the movement direction, placement device position, and score of the moving object group 3000. The control unit 150 associates the generated moving object group information D30, image information D10, and installation state information D40 and stores them in the storage unit 140, and then proceeds to the process of step S103.

[0094] The control unit 150 transmits the moving object group information D30 and the installation state information D40 to the information processing device 200 via the communication unit 130 (step S103). For example, when the control unit 150 generates the moving object group information D30 based on the image information D10, the control unit 150 transmits the moving object group information D30 to the information processing device 200. As a result, the camera 100 can cause the information processing device 200 to track the moving moving object group 3000 by transmitting the moving object group information D30 at the timing when the moving object group 3000 is imaged. When the processing of step S103 ends, the control unit 150 ends the processing procedure shown in FIG. 8.

[0095] 8 can be changed to a processing procedure in which the camera 100 does not transmit the installation state information D40 when the information processing device 200 pre-stores the installation state information D40 of the camera 100. In this case, step S103 may be configured to transmit only the moving object group information D30 to the information processing device 200.

[0096] (Example of processing procedure of information processing device) Fig. 9 is a flowchart showing an example of a processing procedure executed by the information processing device 200. The processing procedure shown in Fig. 9 is realized by the control unit 240 of the information processing device 200 executing the program 231. The processing procedure shown in Fig. 9 is repeatedly executed by the control unit 240.

[0097] 9, the control unit 240 of the information processing device 200 acquires moving object group information D30 and installation state information D40 from the multiple cameras 100 (step S201). For example, the control unit 240 acquires the moving object group information D30 and installation state information D40 for the same day and the same time period received from the multiple cameras 100 via the communication unit 220, and stores them in association with each other in the storage unit 230. When the process of step S201 ends, the control unit 240 advances the process to step S202.

[0098] The control unit 240 compares the multiple moving object groups 3000 (step S202). For example, the control unit 240 inputs the multiple moving object group information D30 to a determination model M2, and stores the output of the determination model M2 in the storage unit 230 as a determination result as to whether the multiple moving object group information D30 belong to the same moving object group 3000. For example, the control unit 240 may compare the multiple moving object group information D30 with each other and determine whether they belong to the same moving object group 3000, without using the determination model M2. When the process of step S202 ends, the control unit 240 advances the process to step S203.

[0099] Based on the comparison result of step S202, the control unit 240 determines whether or not the moving objects 3000 are the same moving object group 3000 (step S203). If the control unit 240 determines that the moving objects 3000 captured by each of the cameras 100 are not the same moving object group 3000 (No in step S203), the control unit 240 proceeds to step S206, which will be described later. If the control unit 240 determines that the moving objects 3000 are the same moving object group 3000 (Yes in step S203), the control unit 240 proceeds to step S204.

[0100] The control unit 240 estimates the installation relationship of the multiple cameras 100 (step S204). For example, the control unit 240 inputs the moving object group information D30 and the installation state information D40 of the multiple moving objects determined to be the same moving object group 3000 to the installation relationship estimation model M20. When the installation relationship estimation model M20 outputs the estimation result, the control unit 240 stores the estimation result in the storage unit 230 as the installation relationship of the multiple cameras 100. When the process of step S204 ends, the control unit 240 proceeds to step S205.

[0101] The control unit 240 outputs the installation relationship information D100 (step S205). For example, the control unit 240 generates the installation relationship information D100 as shown in FIG. 4 based on the estimation result of step S205, and executes an output process of the installation relationship information D100. The output process includes, for example, a process of displaying the installation relationship information D100 on a display device, a process of transmitting the installation relationship information D100 to an electronic device external to the information processing device, etc. When the process of step S205 ends, the control unit 240 advances the process to step S206.

[0102] The control unit 240 determines whether the number of processed moving object groups 3000 is a specified number (step S206). For example, the control unit 240 determines that the number of processed moving object groups 3000 is the specified number when the number of times the installation relationship of the multiple cameras has been estimated or the number of times the installation relationship information D100 has been output reaches a preset specified number. The specified number can be set to, for example, a number of times so as to improve estimation accuracy. If the control unit 240 determines that the number of processed moving object groups 3000 is not the specified number (No in step S206), the control unit 240 returns the process to step S201, which has already been described, and continues the process. If the control unit 240 determines that the number of processed moving object groups 3000 is the specified number (Yes in step S206), the control unit 240 ends the processing procedure shown in FIG. 9.

[0103] (Example of camera system operation) Fig. 10 is a diagram showing an example of the operation of the camera 100 and the information processing device 200 in the camera system 1. In the example shown in Fig. 10, it is assumed that the first camera 100A and the second camera 100B are installed so as to capture images of different imaging areas in the same moving direction 1100M of the same road 1100, as shown in Fig. 1.

[0104] 10, when first camera 100A captures an image of multiple moving objects 2000 moving in its own imaging area, it identifies moving object group 3000 from image information D10 (step S1101). After identifying moving object group 3000, first camera 100A transmits moving object group information D30 via communication unit 130 (step S1102). In this embodiment, first camera 100A transmits moving object group information D30 and installation state information D40 to information processing device 200.

[0105] Furthermore, when the plurality of moving objects 2000 captured by the first camera 100A move in the movement direction 1100M, they enter the imaging area of ​​the second camera 100B.

[0106] When second camera 100B captures an image of multiple moving objects 2000 moving within its own imaging area, it identifies moving object group 3000 from image information D10 (step S1201). After identifying moving object group 3000, second camera 100B transmits moving object group information D30 via communication unit 130 (step S1202). In this embodiment, second camera 100B transmits moving object group information D30 and installation state information D40 to information processing device 200.

[0107] When information processing device 200 receives moving object group information D30 from first camera 100A and then from second camera 100B, information processing device 200 determines whether or not the multiple moving object groups 3000 are the same moving object group 3000 based on the moving object group information D30. Information processing device 200 then determines that the multiple moving object groups 3000 are the same moving object group 3000 (step S1301). Information processing device 200 estimates installation relationship information D100 and outputs the installation relationship information D100 (step S1302).

[0108] This allows the information processing device 200 to output installation relationship information D100 based on how the group of moving objects 3000 appears as indicated by image information D10 captured by the multiple cameras 100. Furthermore, even if the first camera 100A and the second camera 100B are installed at different locations, the information processing device 200 can output installation relationship information D100 based on how the group of moving objects 3000 appears. As a result, even if the multiple cameras are installed near the road 1100, the information processing device 200 can support estimation of the installation status of the multiple cameras 100 capturing images of the traffic environment 1000 without requiring any manual work or traffic regulations. Furthermore, the information processing device 200 can perform calibration for determining the installation status of the cameras 100 without any jigs or work, even when multiple cameras are installed.

[0109] Thereafter, first camera 100A and second camera 100B transmit moving object group information D30 to information processing device 200 every time they capture an image of a moving object group 3000. Then, information processing device 200 outputs installation relationship information D100 every time it determines that a plurality of moving object groups 3000 are the same moving object group 3000. In this way, information processing device 200 can improve the accuracy of estimating the installation relationship by estimating the installation relationship between the plurality of cameras 100 every time a different plurality of moving object groups 3000 pass by first camera 100A and second camera 100B.

[0110] Furthermore, the information processing device 200 ends the operation when the determined number of the moving object group 3000 becomes equal to or greater than a specified number. This allows the information processing device 200 to appropriately end the estimation of the installation relationship of the multiple cameras 100.

[0111] The information processing device 200 can determine whether the moving objects 2000 included in the group 3000 are the same group 3000 based on at least one of the type, arrangement (traveling order), color, movement direction, arrangement, and device position of the moving objects 2000. This allows the information processing device 200 to accurately identify multiple moving object groups 3000, thereby improving the accuracy of determining whether the moving objects are the same group 3000. For example, by focusing on moving object groups 3000 that are a combination of moving objects 2000 of different types, colors, etc., the information processing device 200 can more accurately identify multiple moving object groups 3000 even when multiple cameras 100 are located at different positions.

[0112] The information processing device 200 can output statistical installation relationship information D100 based on the time series information of the first moving object group and the time series information of the second moving object group via the output unit 243. For example, the information processing device 200 can contribute to understanding the movement status of the moving object group 3000, the distance between the cameras 100, etc. by outputting the installation relationship information D100 for each of the multiple cameras 100, in which the date and time when the moving object group 3000 was imaged is associated with the camera 100.

[0113] The information processing device 200 may express the ease of use of the moving object group 3000 as the value of the group itself in estimating the positional relationship between the multiple cameras 100. For example, the information processing device 200 can set a score such that the value of the moving object group 3000 increases as the number of moving objects 2000, the number of types, etc. included in the moving object group 3000 increases. This allows the information processing device 200 to improve the accuracy of determining whether the moving objects are the same moving object group 3000 by not using moving object groups 3000 with low scores in the determination.

[0114] The acquisition unit 241 of the information processing device 200 can acquire first moving object group information and second moving object group information from multiple cameras 100 that are installed so that their imaging areas do not overlap. This allows the information processing device 200 to output installation relationship information D100 for cameras 100 that are installed over a wide area of ​​the road 1100, thereby improving the efficiency of maintenance of the multiple cameras 100.

[0115] (Other embodiments) Fig. 11 is a diagram showing an example of another configuration of the information processing device 200 according to the embodiment. As shown in Fig. 11, the information processing device 200 includes the above-described input unit 210, communication unit 220, storage unit 230, and control unit 240. The control unit 240 may include the above-described acquisition unit 241, determination unit 242, output unit 243, and identification unit 244.

[0116] In this case, the above-described acquisition unit 241 may be configured to have a function of acquiring image information D10 from a plurality of cameras 100. Furthermore, the storage unit 230 may be configured to store the above-described object estimation model M1.

[0117] The identification unit 244 uses the above-mentioned object estimation model M1, object identification program, etc. to identify a moving object group 3000 including one or more moving objects 2000 from the image information D10. The identification unit 244 identifies the moving direction, origin (center) of the moving object group 3000, etc. of the moving object group 3000 based on the presence or absence, type, color, moving direction, arrangement, relative position, score, etc. of the moving object 2000. The identification unit 244 generates moving object group information D30 capable of identifying the identified moving object group 3000, and stores the moving object group information D30 in the storage unit 230 in association with the camera 100.

[0118] The above-described determination unit 242 may be configured to have a function of determining whether or not the plurality of moving object groups 3000 identified by the identification unit 244 are the same moving object group 3000 .

[0119] Fig. 12 is a flowchart showing an example of another processing procedure executed by the information processing device 200. The processing procedure shown in Fig. 12 is a processing procedure obtained by partially modifying the processing procedure shown in Fig. 9. The processing procedure shown in Fig. 12 is repeatedly executed by the control unit 240.

[0120] 12, the control unit 240 of the information processing device 200 acquires image information D10 and installation state information D40 from a plurality of cameras 100 (step S211). For example, the control unit 240 acquires image information D10 and installation state information D40 for the same day and the same time period received from a plurality of cameras 100 via the communication unit 220, and stores the information in association with each other in the storage unit 230. When the process of step S211 ends, the control unit 240 advances the process to step S212.

[0121] The control unit 240 identifies the moving object 2000 from the image information D10 for each of the multiple cameras 100 and generates moving object group information D30 (step S212). For example, the control unit 240 inputs the image information D10 to the object estimation model M1 and identifies the moving object 2000 indicated by the image information D10 based on the output of the object estimation model M1. Then, based on the identification results of the multiple identified moving objects 2000, the control unit 240 generates moving object group information D30 that can identify information such as the type, color, and moving direction of the moving object 2000, as well as the moving direction, placement device position, score, etc. of the moving object group 3000. The control unit 240 associates the generated moving object group information D30, image information D10, and installation state information D40 and stores them in the storage unit 230, and then proceeds to the process already described step S102. The control unit 240 then executes the processes of steps S102 to S106. The processing from step S102 to step S106 is the same as the processing from step S102 to step S106 in FIG. 9, and therefore a description thereof will be omitted.

[0122] In this way, the information processing device 200 according to another embodiment can output the installation relationship information D100 based on how the group of moving objects 3000 appears as indicated by the image information D10 acquired from the multiple cameras 100. Furthermore, even if the multiple cameras 100 are separated from each other, the information processing device 200 can output the installation relationship information D100 based on how the group of moving objects 3000 appears. As a result, even if the multiple cameras are installed near the road 1100, the information processing device 200 can assist in estimating the installation state of the multiple cameras 100 capturing the traffic environment 1000 without requiring any human work or traffic regulations.

[0123] Although the above-described information processing device 200 is described as being provided outside the Fukukaze camera 100, the present invention is not limited to this. For example, the information processing device 200 may be incorporated into one of the multiple cameras 100 and realized by the control unit 150, module, etc. of the camera 100. For example, the information processing device 200 may be incorporated into a traffic light, lighting equipment, communication equipment, etc. installed in the traffic environment 1000.

[0124] The above-described information processing device 200 may be realized by a server device, etc. For example, the information processing device 200 may be a server device that acquires image information D10 from each of the multiple cameras 100, estimates installation state parameters from the image information D10, and provides the estimation results.

[0125] The learning device 300 described above generates the object estimation model M1, the determination model M2, the state estimation model M10, and the installation relationship estimation model M20, but is not limited to this. For example, the learning device 300 may be implemented as being incorporated into the information processing device 200.

[0126] Furthermore, the present disclosure may be applied not only to cases where the object estimation model M1, the judgment model M2, the state estimation model M10, and the installation relationship estimation model M20 are implemented using separate models and separate learning units, but also to an embodiment in which multiple models are combined into an integrated model and machine learning is performed using a single integrated machine learning unit.

[0127] Although specific embodiments have been described to fully and clearly disclose the claimed technology, the appended claims should not be limited to the above-described embodiments, but should be construed to embody all modifications and alternative arrangements that may be made by those skilled in the art within the scope of the basic concept presented herein.

[0128] (Appendix 1) a first identification unit that identifies a first moving object group including at least one moving object from a first image captured by a first camera; a second identification unit that identifies a second moving object group including at least one of the moving objects from a second image captured by a second camera; a determination unit that determines whether the first moving body group and the second moving body group are the same moving body group; an output unit that outputs installation relationship information that can identify the installation relationship between the first camera and the second camera based on the movement pattern of the first moving body group and the movement pattern of the second moving body group that are determined to be the same moving body group; An information processing device comprising: (Appendix 2) Multiple cameras and A camera system including a plurality of cameras and an information processing device capable of communicating with the cameras, The information processing device includes: a determination unit that determines whether a first moving object group identified from a first image captured by a first camera and a second moving object group identified from a second image captured by a second camera are the same moving object group; an output unit that outputs installation relationship information that can identify the installation relationship between the first camera and the second camera based on the movement pattern of the first moving body group and the movement pattern of the second moving body group that are determined to be the same moving body group; A camera system comprising: [Explanation of symbols]

[0129] 1 camera system 100 cameras 100A 1st camera 100B Second Camera 110 Imaging unit 120 Sensor unit 130 Communications Department 140 Storage section 141 Programs 150 control section 151 Image Recognition Unit 152 Identification unit 153 Estimation Department 154 Transmitter 200 Information processing device 210 Input section 220 Communications Department 230 Storage section 231 Programs 232 Setting Information 240 Control Unit 241 Acquisition Department 242 Judgment section 243 Output Section 244 Identification Unit 300 Learning Device 310 Display section 320 Control unit 330 Communications Department 340 Storage section 341 Programs 342 Teacher information 350 control section 351 First acquisition part 352 Machine Learning Department 1 353 Second Acquisition Department 354 Second Machine Learning Department 355 Third Acquisition Department 356 Third Machine Learning Department 357 4th Acquisition Department 358 4th Machine Learning Department 1000 Traffic environment 1100 Road 2000 Mobile 3000 Mobile Group 3100 First Mobile Group 3200 Second Mobile Group D10 Image Information D30 Mobile Group Information D40 Installation status information D100 Installation Information M1 object estimation model M2 decision model M10 State Estimation Model M20 Installation Relationship Estimation Model

Claims

1. a determination unit that determines whether a first moving object group identified from a first image captured by the first camera and a second moving object group identified from a second image captured by the second camera are the same moving object group; an output unit that outputs installation relationship information that can identify the installation relationship between the first camera and the second camera based on the movement pattern of the first moving body group and the movement pattern of the second moving body group that are determined to be the same moving body group; An information processing device comprising:

2. an acquisition unit that acquires first moving object group information indicating the first moving object group including at least two or more moving objects identified from the first image, and second moving object group information indicating the second moving object group including at least two or more moving objects from the second image, the determination unit determines whether the first moving body group indicated by the first moving body group information and the second moving body group indicated by the second moving body group information are the same moving body group. The information processing device according to claim 1 .

3. The determination unit a first movement direction as the movement mode of the first moving body group; determining a second movement direction as the movement mode of the second moving object group; The output unit outputting the installation relationship information capable of identifying an installation relationship between the first camera and the second camera based on the first movement direction and the second movement direction; The information processing device according to claim 2 .

4. The determination unit determining whether the first moving body group and the second moving body group are the same moving body group based on type information of the moving body of the first moving body group and type information of the moving body of the second moving body group; The information processing device according to claim 3 .

5. The determination unit determining whether the first moving body group and the second moving body group are the same moving body group based on the type information of the moving bodies and the location information of the moving bodies of the first moving body group and the type information of the moving bodies and the location information of the moving bodies of the second moving body group; The information processing device according to claim 4 .

6. The determination unit determining whether the first moving body group and the second moving body group are the same moving body group based on the type information and color information of the moving bodies of the first moving body group and the type information and color information of the moving bodies of the second moving body group; The information processing device according to claim 4 .

7. The determination unit determining the type information of the moving objects in the first group of moving objects based on a first size of each moving object relative to the size of the first group of moving objects; determining the type information of the moving objects in the second group of moving objects based on a second size of each of the moving objects relative to the size of the second group of moving objects; The information processing device according to claim 4 .

8. The type information includes at least a standard vehicle, a large vehicle, and a motorcycle. The information processing device according to claim 4 .

9. the determination unit determines whether the first moving body group and the second moving body group are the same moving body group based on a first relative position of each of the moving bodies with the center of gravity of the first moving body group as an origin and a second relative position of each of the moving bodies with the center of gravity of the second moving body group as an origin. The information processing device according to claim 3 .

10. the output unit outputs the statistical installation relationship information based on time-series information on the movement patterns of the first moving body group and time-series information on the movement patterns of the second moving body group. The information processing device according to claim 2 .

11. the output unit outputs the installation relationship information that enables identification of scores of the movement patterns of the first moving object group and the second moving object group. The information processing device according to claim 10.

12. the output unit outputs the installation relationship information capable of identifying the score, which increases as the number of the moving objects or the number of pieces of type information included in the moving object group increases. The information processing device according to claim 11.

13. the determination unit determines whether the first moving body group indicated by the first moving body group information and the second moving body group indicated by the second moving body group information are the same moving body group when the scores of the first moving body group and the second moving body group satisfy a determination condition. The information processing device according to claim 11.

14. the output unit terminates its operation when the number of moving object groups for which it has determined whether a first moving object group identified from a first image captured by the first camera and a second moving object group identified from a second image captured by the second camera are the same moving object group becomes equal to or greater than a specified number. The information processing device according to claim 1 .

15. the output unit outputs the installation relationship information that enables identification of a relationship between the first camera and the second camera. The information processing device according to claim 1 .

16. the acquisition unit acquires the first moving object group information and the second moving object group information from the first camera and the second camera that are installed so that their imaging areas do not overlap. The information processing device according to claim 2 .

17. The acquisition unit: obtaining eleventh moving object information indicating one first moving object identified from the first image and twenty-second moving object information indicating one second moving object from the second image; the determination unit determines whether the first moving object indicated by the eleventh moving object information and the second moving object indicated by the twenty-second moving object information are the same moving object. The information processing device according to claim 2 .

18. The computer determining whether a first moving object group identified from a first image captured by a first camera and a second moving object group identified from a second image captured by a second camera are the same moving object group; The computer outputting installation relationship information capable of identifying an installation relationship between the first camera and the second camera based on the movement patterns of the first moving body group and the second moving body group determined to be the same moving body group; An information processing method, including:

19. On the computer, determining whether a first moving object group identified from a first image captured by a first camera and a second moving object group identified from a second image captured by a second camera are the same moving object group; outputting installation relationship information capable of identifying an installation relationship between the first camera and the second camera based on the movement patterns of the first moving body group and the second moving body group determined to be the same moving body group; An information processing program that executes the above.

Images