Information processing apparatus, control method, program, and imaging system

Abstract

To output three-dimensional shape information according to the state of an imaging apparatus.SOLUTION: An information processing apparatus has: first acquisition means that acquires three-dimensional shape information of a scene picked up by an imaging apparatus; second acquisition means that acquires state information related to the imaging state of the imaging apparatus; output means that outputs the three-dimensional shape information to the imaging apparatus; and control means that controls the operation of the output means. The first acquisition means acquires, for the scene, at least one of first three-dimensional shape information and second three-dimensional shape information with higher spatial resolution than the first three-dimensional shape information. The control means controls, according to the state information, whether to output the first three-dimensional shape information or the second three-dimensional shape information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method, a program, and an imaging system, and particularly to an imaging technique using three-dimensional shape information.

Background Art

[0002] There is an apparatus for generating information on the three-dimensional shape (three-dimensional shape information) of an object. For example, in a distance measuring device such as a LiDAR (Light Detection And Ranging) sensor, the distance from the device to the object is measured, and the three-dimensional shape information of the object can be generated by converting the measurement result.

[0003] Three-dimensional shape information is used not only for applications as a three-dimensional model of an object but also in various technologies. For example, in the field of imaging devices and the like, three-dimensional shape information can be used for detecting a subject when performing imaging control such as focus control and exposure control.

[0004] By the way, the higher the spatial resolution of three-dimensional shape information, the more accurately the shape of the object can be represented. That is, the higher the accuracy of the three-dimensional shape information, the more appropriate imaging control can be performed for the object. On the other hand, since the amount of information of the three-dimensional shape information increases as the spatial resolution increases, the computational load for processing the three-dimensional shape information in the imaging device becomes high. Therefore, Patent Document 1 discloses a technique of performing distance measurement limited to a region estimated to be a human body.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, from the viewpoint of suitably recognizing the scene to be imaged for imaging control, the three-dimensional shape information obtained by distance measurement limited to a partial region as in Patent Document 1 may not be suitable.

[0007] The present invention has been made in view of the above problems, and an object thereof is to provide an information processing apparatus, a control method, a program, and an imaging system that output three-dimensional shape information according to the state of an imaging apparatus.

Means for Solving the Problems

[0008] In order to achieve the above object, an information processing apparatus according to the present invention includes a first acquisition unit that acquires three-dimensional shape information of a scene imaged by an imaging apparatus, a second acquisition unit that acquires state information regarding the imaging state of the imaging apparatus, an output unit that outputs the three-dimensional shape information to the imaging apparatus, and a control unit that controls the operation of the output unit. The first acquisition unit acquires at least either first three-dimensional shape information or second three-dimensional shape information having a higher spatial resolution than the first three-dimensional shape information for the scene, and the control unit controls which of the first three-dimensional shape information and the second three-dimensional shape information is to be output according to the state information.

Effects of the Invention

[0009] According to the present invention having such a configuration, it is possible to output three-dimensional shape information according to the state of the imaging apparatus.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0011] [Embodiment 1] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0012] One embodiment described below is an example of an imaging system including an imaging device 200 and a distance measuring device 110, and a PC 100 capable of outputting three-dimensional shape information of a scene imaged by the imaging device 200. An example in which the present invention is applied to the configured system will be described. However, the present invention is applicable to any device that is connected to the imaging device 200 and the distance measuring device 110, can acquire three-dimensional shape information based on the distance measurement result of the scene, and can output the three-dimensional shape information to the imaging device. Such devices can include, in addition to a PC, various devices having information processing functions such as a tablet computer, a media player, a PDA, a smartphone, a game machine, a robot, a drone, and a drive recorder.

[0013] 《Functional Configuration of Imaging System》 FIG. 1 is a block diagram showing each device included in the imaging system according to the present embodiment and the functional configuration of the PC 100 among them.

[0014] The distance measurement device 110 measures the distance to the scene imaged by the imaging device 200 and generates three-dimensional shape information of the scene. The distance measurement device 110 can employ, for example, a LiDAR sensor. The LiDAR sensor has a light beam output unit that outputs a light beam and irradiates the surface of an object with the light beam, and a reception unit that receives the reflected light beam from the surface of the target object. Then, the LiDAR sensor derives the distance to the object surface in the irradiation direction using the time until the reflected light beam returns or the phase difference between the irradiated light beam and the reflected light beam (distance measurement). By combining the distance regarding the irradiation direction obtained in this way with the information on the irradiation direction, the three-dimensional coordinates of the point on the object surface in the irradiation direction can be specified. The LiDAR sensor can generate a three-dimensional point cloud for the objects distributed in the scene by performing such distance measurement while changing the irradiation direction. Here, since the three-dimensional point cloud generated by the LiDAR sensor can indicate the three-dimensional shape of the object surface existing in the scene, in the following description, the data generated by the LiDAR sensor as the distance measurement result is referred to as "three-dimensional shape information". Therefore, the distance measurement device 110 outputs three-dimensional shape information as the distance measurement result of the scene.

[0015] In addition, in the present embodiment, the distance measurement device 110 will be described as a configuration included in the PC 100, but the implementation of the present invention is not limited to this. Needless to say, the distance measurement device 110 may be an external device configured to be detachable from the PC 100.

[0016] The three-dimensional shape information output by the distance measurement device 110 is stored in the shape information storage unit 122 of the non-volatile memory 102. The non-volatile memory 102 is an electrically erasable and recordable storage device such as an EEPROM, for example. The non-volatile memory 102 is configured to be capable of storing permanent information, and in the present embodiment, its storage area is divided into a system storage unit 121 and a shape information storage unit 122.

[0017] The control unit 101 is a control device having at least one processor, and controls the operations of each block of the information processing device 100. Specifically, the control unit 101 controls the operations of each block by expanding and executing the operation programs of each block stored in the system storage unit 121 of the non-volatile memory 102 described later in the system memory 103. In the PC 100 of the present embodiment, the control unit 101 also operates as a generation unit 131, an output control unit 132, and an acquisition unit 133 by executing the operation program.

[0018] Based on the three-dimensional shape information output by the distance measuring device 110, the generation unit 131 generates three-dimensional shape information with a higher spatial resolution than the three-dimensional shape information. In the following description, in order to distinguish between the three-dimensional shape information output by the distance measuring device 110 and the three-dimensional shape information generated by the generation unit 131, the former is referred to as "low-resolution shape information" and the latter is referred to as "high-resolution shape information". The generation of high-resolution shape information from low-resolution shape information may be performed, for example, by using an inference model obtained by deep learning using a CNN. The high-resolution shape information generated by the generation unit 131 is stored in the shape information storage unit 122 of the non-volatile memory 102.

[0019] The output control unit 132 controls the output of the three-dimensional shape information to the imaging device 200. As described above, in the PC 100 of the present embodiment, two types of three-dimensional shape information, low-resolution shape information and high-resolution shape information, can be output to the imaging device 200. Therefore, the output control unit 132 controls which three-dimensional shape information to output.

[0020] The acquisition unit 133 acquires information indicating the imaging state of the imaging device 200 (hereinafter referred to as state information). In the present embodiment, the PC 100 intermittently monitors the connected imaging device 200 and sequentially acquires the state information. The state information may be intermittently sent and acquired from the imaging device 200, or may be returned from the imaging device 200 in response to an acquisition request from the acquisition unit 133.

[0021] These functional configurations may be realized by a single processor constituting the control unit 101, or may be realized by a plurality of processors working together or being shared among a plurality of processors.

[0022] The system memory 103 is a rewritable volatile storage device such as a DRAM. The system memory 103 is used not only as a deployment area for the operation program of each block, but also as a storage area for information such as intermediate data output in the operation of each block.

[0023] The imaging device 200 is, for example, a digital still camera. The imaging device 200 includes a lens unit (imaging optical system), an imaging element that converts an optical image of a scene to be imaged into an analog image signal, and an A / D converter that converts the analog image signal into a digital signal. An optical image is input to the imaging element via the lens unit, and image data is acquired by converting the electrical signal converted by the imaging element into a digital signal. The image data acquired by the imaging device 200 through imaging is recorded, for example, on a recording medium (not shown) detachably connected to the imaging device 200.

[0024] As described above, in the imaging system of this embodiment, the imaging device 200 acquires the three-dimensional shape information output by the PC 100. Here, the three-dimensional shape information output by the PC 100 is at least one of low-resolution shape information and high-resolution shape information. The imaging device 200 is configured to be able to execute a predetermined operation using the acquired three-dimensional shape information regardless of which information is acquired. At this time, since the amount of information of the low-resolution shape information is less than that of the high-resolution shape information, a predetermined operation can be executed with a smaller computational load using the former than using the latter. On the other hand, since the latter has a higher spatial resolution and shows the shape of the objects distributed in the scene imaged by the imaging device 200 in more detail, the reliability and accuracy of the predetermined operation will increase when using the latter rather than the former.

[0025] In the imaging device 200, a predetermined operation using the three-dimensional shape information can include, for example, an operation of changing the focusing state of the optical system. In this case, since it is possible to grasp the distribution of objects in the scene based on the three-dimensional shape information, the distance from the imaging device 200 to each object can be obtained. Therefore, for example, in response to an operation of designating (designating in the captured image) the subject to be focused on in the imaging device 200, the subject distance of the subject is determined as the focus position of the imaging optical system based on the three-dimensional shape information. Alternatively, for example, the subject distances of the objects distributed in a predetermined region within the imaging angle of view of the imaging device 200 are specified based on the three-dimensional shape information, and the subject distance is determined as the focus position of the imaging optical system. Thereafter, control is performed to change the state of the imaging optical system so as to focus on the determined focus position in the imaging device 200.

[0026] Also, a predetermined operation using the three-dimensional shape information executed in the imaging device 200 can include, for example, an exposure control operation according to the subject. In this case, for example, an object showing a predetermined shape is detected as a subject based on the three-dimensional shape information, and exposure control is performed so that the subject appears in the image data with appropriate exposure.

[0027] In this embodiment, two examples of the predetermined operation using the three-dimensional shape information executed in the imaging device 200 have been given, but it will be easily understood that the implementation of the present invention is not limited to these. That is, in the imaging device 200, the three-dimensional shape information may be used in other modes.

[0028] 《Outline of Output Control》 Hereinafter, the output control of the three-dimensional shape information performed by the output control unit 132 will be described in detail.

[0029] In the PC 100 of this embodiment, the output control unit 132 basically outputs the high-resolution shape information generated by the generation unit 131 to the imaging device 200 based on the low-resolution shape information acquired from the distance measuring device 110. On the other hand, since the high-resolution shape information has a larger amount of information than the low-resolution shape information, the calculation load becomes higher when used for a predetermined operation in the imaging device 200. Such an increase in the calculation load may also affect the shooting in the imaging device 200. As a result, there is a risk that a suitable shooting operation such as the image data of the desired image quality is not recorded or the shooting is not performed at the desired timing.

[0030] In this embodiment, as the state of the imaging device 200 in which an increase in the load caused by the use of such high-resolution shape information should be avoided, a state in which the imaging device 200 is in continuous shooting is defined. Therefore, when continuous shooting is being performed in the imaging device 200, the PC 100 of this embodiment is configured not to output high-resolution shape information to the imaging device 200. That is, the output control unit 132 controls to output low-resolution shape information as the three-dimensional shape information if the imaging device 200 is in continuous shooting, and to output high-resolution shape information if it is not in continuous shooting.

[0031] To realize such output control, in the imaging system of this embodiment, the acquisition unit 133 intermittently acquires state information from the imaging device 200. The state information includes, in this embodiment, at least information that can identify whether continuous shooting is being executed in the imaging device 200. The output control unit 132 determines whether continuous shooting is being executed in the imaging device 200 based on the state information acquired by the acquisition unit 133. The output control unit 132 controls which of the low-resolution shape information and the high-resolution shape information is output as the three-dimensional shape information based on whether continuous shooting is being executed in the imaging device 200.

[0032] Here, the low-resolution shape information output by the distance measurement device 110 can change from moment to moment. If the high-resolution shape information is not output to the imaging device 200, there is no need for the PC 100 to perform processing to generate high-resolution shape information from the low-resolution shape information. Therefore, when the output control unit 132 determines that continuous shooting is being performed in the imaging device 200, it controls the generation unit 131 not to perform the generation process of the high-resolution shape information. In other words, in the mode of directly outputting the low-resolution shape information acquired from the distance measurement device 110, the output control unit 132 controls the generation unit 131 not to perform the generation process of the high-resolution shape information. That is, in the imaging system of the present embodiment, when continuous shooting is being performed in the imaging device 200, the generation of high-resolution shape information is not performed in the PC 100, and the output of the low-resolution shape information acquired from the distance measurement device 110 is controlled to be performed.

[0033] 《Output Control Process》 Hereinafter, the output control process executed in the PC 100 of the present embodiment will be specifically described using the flowchart of FIG. 2. The process corresponding to the flowchart can be realized by the control unit 101 reading out the corresponding processing program stored in the non-volatile memory 102, for example, and expanding and executing it in the system memory 103. This output control process is started, for example, when it is detected that the imaging device 200 and the PC 100 are connected and a shooting mode in which a predetermined operation using the three-dimensional shape information is set in the imaging device 200, and will be described as being repeatedly executed at a predetermined frequency. In the following description, when the operations of each step are realized by the functional configurations (generation unit 131, output control unit 132, acquisition unit 133) of the control unit 101, they will be described in a manner centered on the functional configurations.

[0034] In S201, the acquisition unit 133 acquires the status information from the imaging device 200.

[0035] In S202, the output control unit 132 determines whether continuous shooting is being executed in the imaging device 200 based on the state information acquired in S201. If the output control unit 132 determines that continuous shooting is being executed in the imaging device 200, the process proceeds to S203, and if it determines that continuous shooting is not being executed, the process proceeds to S204.

[0036] In S203, the output control unit 132 outputs the low-resolution shape information stored in the shape information storage unit 122 to the imaging device 200 as three-dimensional shape information to complete this output control process.

[0037] On the other hand, when it is determined in S202 that continuous shooting is being executed, in S204, the output control unit 132 causes the generation unit 131 to generate high-resolution shape information. More specifically, the generation unit 131 generates high-resolution shape information based on the low-resolution shape information stored in the shape information storage unit 122 and stores it in the shape information storage unit 122.

[0038] In S205, the output control unit 132 outputs the high-resolution shape information generated in S204 to the imaging device 200 as three-dimensional shape information to complete this output control process.

[0039] As described above, according to the information processing apparatus of the present embodiment, three-dimensional shape information corresponding to the state of the imaging device can be output. More specifically, when continuous shooting is being executed in the imaging device, the information processing apparatus of the present embodiment reduces the computational load generated in the imaging device by outputting low-resolution shape information to the imaging device, and can ensure suitable recording of image data by continuous shooting. On the other hand, when continuous shooting is not being executed in the imaging device, the information processing apparatus can improve the accuracy of focus control and subject detection in the imaging device by outputting high-resolution shape information to the imaging device.

[0040] In addition, in this embodiment, an aspect has been described in which information indicating whether continuous shooting is being executed in the imaging device 200 is explicitly included in the state information. However, the implementation of the present invention is not limited to this. Whether continuous shooting is being executed in the imaging device 200 may be specified by, for example, acquiring information on the output frequency of image data from the imaging device 200 or additional information such as the shooting time included in each image data as state information and analyzing it.

[0041] [Modification Example 1] In the above-described embodiment, an aspect has been described in which low-resolution shape information is output over the period during which continuous shooting is actually being executed by including information indicating whether continuous shooting is being executed in the imaging device 200 in the state information. However, the control for outputting low-resolution shape information without outputting high-resolution shape information does not necessarily need to be performed on the condition that continuous shooting is being executed at the time of acquiring the state information. That is, such control may be executed as long as it is executed when a high-frequency shooting operation is performed in the imaging device 200, and various modifications are possible for the method of specifying the presence or absence of the shooting operation.

[0042] For example, control may be performed when the state information indicates that the imaging device 200 is in a state where continuous shooting can be executed, such as when a setting for continuous shooting has been made in the imaging device 200 or when the imaging device 200 is started in a shooting mode for continuous shooting.

[0043] Also, in shooting such as so-called time-lapse shooting even in continuous shooting, the shooting frequency is low, and there may be no increase in the calculation load that affects the shooting process even if high-resolution shape information is used. Therefore, the state information may include information on the shooting frequency, such as information on the interval during continuous shooting or information on the shooting frame rate during video shooting, and the output control unit 132 may perform output control based on the information on the shooting frequency. In one aspect, the output control unit 132 can be controlled to output low-resolution shape information when the shooting frequency exceeds a predetermined threshold and to output high-resolution shape information when it is below the predetermined threshold.

[0044] [Embodiment 2] In the above-described embodiments and modifications, from the viewpoint of avoiding the compression of the computing resources in the imaging device 200, the mode of controlling to output low-resolution shape information when high-frequency imaging processing is executed has been described. However, the output control of the three-dimensional shape information may be performed from the viewpoint of avoiding the occurrence of processing delay, rather than from the viewpoint of avoiding the compression of the computing resources.

[0045] For example, when the imaging device 200 is imaging a specific type of subject that is assumed to move at high speed, a short exposure time is set in order to capture the subject in a suitable state at the timing when the shooting instruction is given by the photographer. That is, if high-resolution shape information is output from the PC 100 in such a situation, calculations for processing the high-resolution shape information are executed in the imaging device 200, so there is a possibility that the imaging operation cannot be started immediately at the timing when the shooting instruction is given. Alternatively, the time from one imaging operation until the next imaging operation can be executed may be prolonged by the processing of the high-resolution shape information, resulting in the possibility of missing the opportunity to capture the subject at the timing when it reaches the desired state.

[0046] In the imaging system of this embodiment, as the state of the imaging device 200 in which the occurrence of processing delay due to the use of high-resolution shape information should be avoided, a state in which the imaging device 200 captures a specific type of subject within the imaging angle is defined. Therefore, when a specific type of subject is captured within the imaging angle in the imaging device 200, the PC 100 of this embodiment is configured not to output high-resolution shape information to the imaging device 200. That is, the output control unit 132 controls to output low-resolution shape information as the three-dimensional shape information if the imaging device 200 captures a specific type of subject within the imaging angle, and to output high-resolution shape information if not.

[0047] In the following, it will be described assuming that the subject type that does not output high-resolution shape information is "person", but it will be easily understood that the implementation of the present invention is not limited to this. That is, the subject type for which a shooting operation with a short exposure time should be performed in response to the shooting instruction as described above can be set based on, for example, the movement tendency of each subject, and can be other subjects such as animals and moving objects that are not people. In one aspect, such a movement tendency can be determined based on information on the recommended shutter speed at the time of imaging, which is determined for each subject type. At this time, the subject type for which the recommended shutter speed is set shorter than a predetermined time can be specified as the type for which a shooting operation with a short exposure time should be performed. Note that the setting of "person" as the subject type that does not output high-resolution shape information is merely an example, and in other aspects, a configuration that controls to output high-resolution shape information when a person is captured in the imaging angle of view is not excluded.

[0048] To realize such output control, in the imaging system of this embodiment, the PC 100 has the functional configuration shown in FIG. 3. In this embodiment, the acquisition unit 133 intermittently acquires the imaging image being imaged in the imaging device 200 as state information. The imaging image acquired by the acquisition unit 133 is stored in the image storage unit 321 of the non-volatile memory 102. The PC 100 of this embodiment also operates as a specifying unit 331, and the specifying unit 331 specifies the type of the subject captured in the imaging image (in the imaging angle of view) based on the imaging image stored in the image storage unit 321. Here, the specification of the subject type by the specifying unit 331 may be performed, for example, by using an inference model obtained by deep learning using a CNN. The information on the subject type specified by the specifying unit 331 is stored in, for example, the system memory 103 so that the output control unit 132 can refer to it.

[0049] The output control unit 132 controls which of the low-resolution shape information and the high-resolution shape information is output to the imaging device 200 based on the subject type specified by the specifying unit 331. That is, in the present embodiment, when the subject type specified by the specifying unit 331 is a person, the output control unit 132 controls to output the high-resolution shape information to the imaging device 200.

[0050] Similar to the above-described Embodiment 1, if the high-resolution shape information is not output to the imaging device 200, it is not necessary to perform the process of generating the high-resolution shape information from the low-resolution shape information in the PC 100. Therefore, when the subject type captured within the imaging angle of view of the imaging device 200 is a person, the output control unit 132 controls the generation unit 131 not to perform the generation process of the high-resolution shape information.

[0051] 《Output Control Process》 Hereinafter, the output control process executed in the PC 100 of the present embodiment will be specifically described using the flowchart of FIG. 4. The process corresponding to the flowchart can be realized by the control unit 101 reading out the corresponding processing program stored in the non-volatile memory 102, for example, and expanding and executing it in the system memory 103. This output control process is described as being started when it is detected that, for example, the imaging device 200 and the PC 100 are connected and a shooting mode in which a predetermined operation using the three-dimensional shape information is executed is set in the imaging device 200, and is repeatedly executed at a predetermined frequency.

[0052] In the following description, when the operations of each step are realized by the functional configuration (generation unit 131, output control unit 132, acquisition unit 133) of the control unit 101, they will be described in a manner centered on the functional configuration. Also, in the description of the output control process of the present embodiment, steps that perform the same processes as the output control process of Embodiment 1 will be given the same reference numerals and the description will be omitted, and the details of the steps that perform the processes specific to the present embodiment will be described in detail.

[0053] In S401, the acquisition unit 133 acquires a captured image from the imaging device 200 and stores it in the image storage unit 321. When the captured image is stored in the image storage unit 321, the identification unit 331 identifies the type of subject captured in the captured image.

[0054] In S402, based on the information on the type of subject identified in S401, the output control unit 132 determines whether a person is captured within the imaging range of the imaging device 200. If the output control unit 132 determines that a person is captured within the imaging range of the imaging device 200, the process proceeds to S203, and if it determines that no person is captured, the process proceeds to S204.

[0055] As described above, according to the information processing apparatus of the present embodiment, the 3D shape information to be output can be varied according to the state of what subject the imaging device has captured within the imaging range. More specifically, when the type of subject being imaged by the imaging device is a predetermined subject type, the information processing apparatus of the present embodiment can suppress the occurrence of processing delay in the imaging device by outputting low-resolution shape information to the imaging device. On the other hand, when the type of subject being imaged by the imaging device is not a predetermined subject type, by outputting high-resolution shape information to the imaging device, it is possible to improve the accuracy of focus control, subject detection, etc. in the imaging device.

[0056] In the present embodiment, an aspect has been described in which a captured image is acquired from the imaging device 200 as state information, and the type of subject is identified in the PC 100 based on the captured image. However, the implementation of the present invention is not limited to this. For example, the identification of the type of subject may be performed in the imaging device 200, and the information on the type of subject may be configured to be acquirable as state information. Alternatively, the identification of the type of subject may be performed based on the detection results of other sensors such as shape information acquired by an ultrasonic sensor, temperature information acquired by a temperature sensor, or a combination thereof.

[0057] Also, in the present embodiment, although the mode of determining whether the subject being imaged by the imaging device 200 is a subject that should perform a shooting operation with a short exposure time based on the subject type has been described, the implementation of the present invention is not limited to this. Whether to perform a shooting operation with a short exposure time can also be determined based on, for example, the movement state of the subject specified based on a group of continuously acquired imaging images. For example, the movement state can be specified based on the amount of movement of the subject's image with respect to the elapsed time corresponding to the group of imaging images, and it may be determined that a shooting operation with a short exposure time should be performed when the subject is moving or moving at high speed.

[0058] [Modification Example 2] In the above-described Embodiment 2, although the mode of varying the three-dimensional shape information output according to whether the subject type is a specific type that should perform a shooting operation with a short exposure time has been described, the implementation of the present invention is not limited to this. That is, without depending on the subject type, when a high-speed shutter speed is set in the imaging device, in order to reduce the occurrence of processing delay in the imaging device 200, the output control unit 132 may control to output low-resolution shape information. In this mode, the state information acquired by the acquisition unit 133 includes information on the shutter speed set in the imaging device 200. Then, the output control unit 132 may control to output low-resolution shape information when the shutter speed is shorter than a predetermined time, and output high-resolution shape information when it is longer than the predetermined time.

[0059] [Modification Example 3] In the above-described embodiment and modification example, although the mode of controlling the generation unit 131 not to generate high-resolution shape information when outputting low-resolution shape information to the imaging device 200 in order to reduce the calculation amount in the PC 100 has been described, the implementation of the present invention is not limited to this. The high-resolution shape information may be generated for all the low-resolution shape information acquired from the distance measuring device 110, for example, to be available for analysis of the image data after shooting.

[0060] [Modification Example 4] In the above-described embodiments and modified examples, the inference model obtained by deep learning using a CNN was used to generate high-resolution shape information based on low-resolution shape information. However, the implementation of the present invention is not limited to this. The generation of high-resolution shape information may be performed, for example, using an inference model obtained by other known deep learning such as a Diffusion generation model. Alternatively, the generation of high-resolution shape information may be performed by adopting other methods that do not utilize deep learning such as the bicubic method. In addition, with reference to the captured image of the imaging device 200 together with the low-resolution shape information, high-resolution shape information may be generated by adopting a method such as Depth Completion.

[0061] [Embodiment 3] In the above-described embodiments and modified examples, the three-dimensional shape information output by the distance measuring device 110 was used as low-resolution shape information, and the mode in which the generation unit 131 generates high-resolution shape information from the low-resolution shape information was described. However, the implementation of the present invention is not limited to this. For example, in a mode in which the distance measuring device 110 is configured to be able to generate both low-resolution shape information and high-resolution shape information, the present invention can be implemented even if the generation unit 131 does not generate high-resolution shape information. That is, all of the three-dimensional shape information output from the PC 100 to the imaging device 200 is generated by the distance measuring device 110, and the control unit 101 performs output control by varying the three-dimensional shape information generated by the distance measuring device 110 according to the state information. In other words, the output control unit 132 of the present embodiment is configured to transmit the three-dimensional shape information acquired from the distance measuring device 110 to the imaging device 200 regardless of the state information.

[0062] To realize such output control, in the imaging system of the present embodiment, the PC 100 has the functional configuration shown in FIG. 5. As shown in the figure, in the PC 100 of the present embodiment, instead of having the functional configuration of the generation unit 131, the control unit 101 has a distance measurement control unit 531 that controls the operation of the distance measuring device 110.

[0063] The distance measurement control unit 531 performs control to change the three-dimensional shape information generated by the distance measurement device 110 according to the state information. The distance measurement device 110 of the present embodiment is configured such that the spatial resolution can be switched in generating the three-dimensional shape information.

[0064] For example, consider a case where the scene measured by the distance measurement device 110 (corresponding to the scene imaged by the imaging device 200) is in the form shown in Fig. 6(a). At this time, in the distance measurement with the spatial resolution of the distance measurement device 110 set to the maximum, a three-dimensional point cloud as shown in Fig. 6(b) can be obtained. On the other hand, in the distance measurement with the spatial resolution of the distance measurement device 110 set to the minimum (limited value), a three-dimensional point cloud as shown in Fig. 6(c) can be obtained. As shown in the figure, the density of the obtained three-dimensional point cloud changes as the spatial resolution of the distance measurement changes. More specifically, the higher the spatial resolution, the denser the three-dimensional point cloud is with respect to the space, and the lower the spatial resolution, the sparser the three-dimensional point cloud is with respect to the space.

[0065] Therefore, the distance measurement device 110 can generate high-resolution shape information from the dense three-dimensional point cloud obtained in the state where the spatial resolution is set to the maximum. Also, the distance measurement device 110 can generate low-resolution shape information from the sparse three-dimensional point cloud obtained in the state where the spatial resolution is set to the minimum.

[0066] In this way, the distance measurement control unit 531 can control the distance measurement device 110 to generate three-dimensional shape information with different spatial resolutions, and as a result, the spatial resolution of the three-dimensional shape information output to the imaging device 200 by the output control unit 132 can be made different. In other words, the distance measurement device 110 is configured to output either low-resolution shape information or high-resolution shape information under the control of the distance measurement control unit 531. That is, similar to the above-described embodiments and modifications, the high-resolution shape information is stored in the shape information storage unit 122 only when outputting to the imaging device 200.

[0067] In addition, in the present embodiment, the acquisition unit 133 intermittently acquires state information including information that can identify whether or not continuous shooting is being executed from the imaging device 200, as in the first embodiment. That is, based on the state information, the output control unit 132 determines whether or not the imaging device 200 is executing continuous shooting, and can cause the distance measurement control unit 531 to switch the operation of the distance measurement device 110.

[0068] 《Output Control Process》 Hereinafter, the output control process executed in the PC 100 of the present embodiment will be specifically described using the flowchart of FIG. 7. The process corresponding to the flowchart can be realized by the control unit 101 reading out a corresponding processing program stored in, for example, the non-volatile memory 102 and expanding and executing it in the system memory 103. This output control process is started, for example, when it is detected that the imaging device 200 and the PC 100 are connected and a shooting mode in which a predetermined operation using three-dimensional shape information is executed is set in the imaging device 200, and will be described as being repeatedly executed at a predetermined frequency.

[0069] In the following description, when the operations of each step are realized by the functional configuration (generation unit 131, output control unit 132, acquisition unit 133) of the control unit 101, they will be described in a manner centered on the functional configuration. Also, in the description of the output control process of the present embodiment, steps that perform the same processing as the output control process of the first embodiment will be given the same reference numerals and the description will be omitted, and details of steps that perform processing specific to the present embodiment will be described.

[0070] If it is determined in S202 that continuous shooting is being executed, the distance measurement control unit 531 controls the operation of the distance measurement device 110 to generate low-resolution shape information in S701 under the control of the output control unit 132. In response to the processing of this step, the distance measurement device 110 performs distance measurement of the scene and outputs low-resolution shape information, and the low-resolution shape information is stored in the shape information storage unit 122.

[0071] On the other hand, when it is determined in S202 that continuous shooting is not being executed, the distance measurement control unit 531 controls the operation of the distance measurement device 110 to generate high-resolution shape information in S702 under the control of the output control unit 132. In response to the processing of this step, the distance measurement device 110 performs distance measurement on the scene and outputs high-resolution shape information, and the high-resolution shape information is stored in the shape information storage unit 122.

[0072] In S703, the output control unit 132 outputs the three-dimensional shape information stored in the shape information storage unit 122 to the imaging device 200 to complete this output control process. That is, when continuous shooting is being executed in the imaging device 200, low-resolution shape information is output, and when continuous shooting is not being executed, high-resolution shape information is output.

[0073] By doing so, three-dimensional shape information corresponding to the state of the imaging device can be output in the same manner as in Embodiment 1. In this embodiment, the state information has the same configuration as in Embodiment 1, and the mode of switching the three-dimensional shape information output according to whether continuous shooting is being executed in the imaging device 200 has been described. However, the implementation of the present invention is not limited to this. The state information can be information capable of specifying the type of subject captured in the imaging range of the imaging device 200 as in Embodiment 2, and the mode of switching the three-dimensional shape information output according to the type of subject can also be adopted.

[0074] [Modification Example 5] In the above-described embodiments and modification examples, a LiDAR sensor is adopted as the distance measurement device 110, and the mode of using the three-dimensional point cloud as the three-dimensional shape information has been described. However, the implementation of the present invention is not limited to this. The distance measurement device 110 is not limited to a LiDAR sensor, and for example, a laser scanner, stereo vision hardware, or a system that generates a three-dimensional shape based on a moving image can also be adopted. Also, the three-dimensional shape information can be information in a format configured by a known method such as voxels, meshes, implicit function representations, etc., instead of the three-dimensional point cloud.

[0075] [Modification Example 6] In the above-described embodiments and modifications, from the perspective of avoiding the compression of computational resources in the imaging device 200 or from the perspective of avoiding the occurrence of processing delays, the mode of restricting the output of high-resolution shape information has been described. However, the implementation of the present invention is not limited to this.

[0076] For example, when the imaging device 200 detects a predetermined subject and performs imaging in a low-luminance scene, it may be difficult to detect the subject based on the captured image of the scene. In such a case, it is more suitable to use the three-dimensional shape information for subject detection. Also, considering that the subject is mainly detected based on the three-dimensional shape information, using high-resolution shape information can ensure the detection accuracy of the subject.

[0077] Therefore, when the imaging device 200 is in a state where it is imaging a low-luminance scene and is required to accurately detect a predetermined subject, the output control unit 132 may control the imaging device 200 to output high-resolution shape information. In one aspect, to realize such control, information on the shooting mode set in the imaging device 200 is included as state information, and the output control unit 132 performs output control based on the information on the shooting mode. For example, for each shooting mode, information on the detection accuracy of the subject required according to the necessity of subject detection or the like is pre-associated, and the output control unit 132 switches the three-dimensional shape information to be output according to the information on the detection accuracy of the subject defined for the set shooting mode. The required detection accuracy is classified into, for example, two levels: a first detection accuracy (no detection accuracy is required) and a second detection accuracy (detection accuracy is required). The output control unit 132 controls to output high-resolution shape information when it is the latter and low-resolution shape information when it is the former.

[0078] By doing so, it is possible to output suitable three-dimensional shape information according to the usage and the like in the imaging device 200.

[0079] [Modification 7] In the above-described embodiments and modifications, the mode of realizing the present invention in an imaging system in which a plurality of devices (imaging device 200, PC 100 (information processing device), and distance measuring device 110) are connected has been described. However, the implementation of the present invention is not limited thereto. It will be easily understood that the present invention can also be implemented, for example, in a mode in which the distance measuring device 110 is incorporated into the information processing device, or in a mode in which the information processing device is incorporated into the imaging device 200 as a processor.

[0080] [Summary of Embodiments and Modifications] The disclosure of this specification includes the following information processing device, control method, program, and imaging system. (Item 1) First acquisition means for acquiring three-dimensional shape information of a scene imaged by an imaging device; Second acquisition means for acquiring state information regarding an imaging state of the imaging device; Output means for outputting the three-dimensional shape information to the imaging device; Control means for controlling the operation of the output means; having The first acquisition means acquires at least one of first three-dimensional shape information and second three-dimensional shape information having a higher spatial resolution than the first three-dimensional shape information for the scene; The control means controls which of the first three-dimensional shape information and the second three-dimensional shape information is to be output according to the state information. An information processing device characterized by the above. (Item 2) The state information includes information regarding a shooting frequency in the imaging device; The control means causes the output means to output the first three-dimensional shape information when the shooting frequency in the imaging device exceeds a predetermined threshold; causes the output means to output the second three-dimensional shape information when the shooting frequency in the imaging device is lower than the predetermined threshold. The information processing device according to Item 1, characterized by the above. (Item 3) The state information includes information indicating whether continuous shooting is being executed in the imaging device. The control means causes the output means to output the first three-dimensional shape information when continuous shooting is being executed in the imaging device. causes the output means to output the second three-dimensional shape information when continuous shooting is not being executed in the imaging device. The information processing apparatus according to item 1 or 2, characterized in that. (Item 4) The state information includes information indicating the type of subject being imaged by the imaging device. The control means causes the output means to output the first three-dimensional shape information when the type of subject being imaged by the imaging device is a predetermined subject type. causes the output means to output the second three-dimensional shape information when the type of subject being imaged by the imaging device is not the predetermined subject type. The information processing apparatus according to any one of items 1 to 3, characterized in that. (Item 5) The information processing apparatus according to item 4, characterized in that the predetermined subject type is a subject type for which the recommended shutter speed at the time of shooting is set shorter than a predetermined time. (Item 6) The state information includes information on the shutter speed set in the imaging device. The control means causes the output means to output the first three-dimensional shape information when the shutter speed set in the imaging device is shorter than a predetermined time. causes the output means to output the second three-dimensional shape information when the shutter speed set in the imaging device is longer than the predetermined time. The information processing apparatus according to any one of items 1 to 5, characterized in that. (Item 7) The state information includes information indicating the shooting mode set in the imaging device. The imaging modes set in the imaging device are each associated with the required detection accuracy of the subject. The control means causes the output means to output the first three-dimensional shape information when an imaging mode requiring the first detection accuracy is set in the imaging device. causes the output means to output the second three-dimensional shape information when an imaging mode requiring a second detection accuracy higher than the first detection accuracy is set in the imaging device. The information processing apparatus according to any one of items 1 to 6, characterized in that. (Item 8) The information processing apparatus further comprises a generation means for generating the second three-dimensional shape information based on the first three-dimensional shape information acquired by the first acquisition means. The first acquisition means acquires the second three-dimensional shape information generated by the generation means. The information processing apparatus according to any one of items 1 to 7, characterized in that. (Item 9) The information processing apparatus according to item 8, characterized in that the control means further controls the generation means not to generate the second three-dimensional shape information when outputting the first three-dimensional shape information to the output means. (Item 10) The information processing apparatus further comprises a distance measuring means for performing distance measurement on the scene to generate the three-dimensional shape information. The first acquisition means acquires the three-dimensional shape information generated by the distance measuring means. The information processing apparatus according to item 8 or 9, characterized in that. (Item 11) The information processing apparatus further comprises a distance measuring means for performing distance measurement on the scene to generate the three-dimensional shape information. The first acquisition means acquires the three-dimensional shape information generated by the distance measuring means. The control means controls the spatial resolution of the distance measurement by the distance measuring means according to the state information. causes the output means to output the three-dimensional shape information acquired by the first acquisition means. The information processing apparatus according to any one of Items 1 to 7, characterized in that... (Item 12) A first acquisition step of acquiring three-dimensional shape information of a scene imaged by an imaging device; A second acquisition step of acquiring state information regarding the imaging state of the imaging device; An output step of outputting the three-dimensional shape information to the imaging device; A control step of controlling the operation in the output step; having in the first acquisition step, at least one of first three-dimensional shape information and second three-dimensional shape information having a higher spatial resolution than the first three-dimensional shape information is acquired for the scene; in the control step, which of the first three-dimensional shape information and the second three-dimensional shape information is to be output is controlled according to the state information A control method for an information processing apparatus, characterized in that... (Item 13) A program for causing a computer to function as each means of the information processing apparatus according to any one of Items 1 to 11. (Item 14) An imaging system including an imaging device, a distance measuring device that measures distance for a scene imaged by the imaging device to generate three-dimensional shape information, and an information processing device that controls the output of the three-dimensional shape information to the imaging device, wherein the information processing device has a first acquisition means for acquiring the three-dimensional shape information from the distance measuring device; a second acquisition means for acquiring state information regarding the imaging state of the imaging device from the imaging device; an output means for outputting the three-dimensional shape information to the imaging device; a control means for controlling the operation of the output means; having the first acquisition means acquires at least one of first three-dimensional shape information and second three-dimensional shape information having a higher spatial resolution than the first three-dimensional shape information for the scene; The control means controls which of the first 3D shape information and the second 3D shape information is to be output according to the state information. An imaging system characterized by the above. (Item 15) The imaging device includes an imaging optical system, imaging means, acquisition means for acquiring the 3D shape information output by the output means, determination means for determining a focus position based on the 3D shape information acquired by the acquisition means, changing means for changing the state of the imaging optical system based on the focus position determined by the determination means, and has The imaging system according to Item 14, characterized by the above. (Item 16) The imaging device includes imaging means, acquisition means for acquiring the 3D shape information output by the output means, detection means for detecting a subject based on the 3D shape information acquired by the acquisition means, exposure control means for performing exposure control of the imaging means based on the detection result by the detection means, and has The imaging system according to Item 14, characterized by the above.

[0081] [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0082] The invention is not limited to the above embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, claims are attached to disclose the scope of the invention.

Description of Signs

[0083] 100: PC, 101: Control Unit, 110: Distance Measuring Device, 131: Generation Unit, 132: Output Control Unit, 133: Acquisition Unit, 200: Imaging Device, 321: Image Memory Unit, 331: Identification Unit, 531: Distance Measurement Control Unit

Claims

1. A first acquisition means for acquiring three-dimensional shape information of a scene imaged by an imaging device; A second acquisition means for acquiring state information regarding the imaging state of the imaging device; An output means for outputting the three-dimensional shape information to the imaging device; A control means for controlling the operation of the output means; characterized by comprising: The first acquisition means acquires at least one of first three-dimensional shape information and second three-dimensional shape information having a higher spatial resolution than the first three-dimensional shape information for the scene; The control means controls which of the first three-dimensional shape information and the second three-dimensional shape information is to be output according to the state information. An information processing apparatus characterized by the above.

2. The state information includes information regarding the shooting frequency in the imaging device, The control means: causes the output means to output the first three-dimensional shape information when the shooting frequency in the imaging device exceeds a predetermined threshold; causes the output means to output the second three-dimensional shape information when the shooting frequency in the imaging device is below the predetermined threshold. The information processing apparatus according to claim 1, characterized by the above.

3. The state information includes information indicating whether continuous shooting is being executed in the imaging device, The control means: causes the output means to output the first three-dimensional shape information when continuous shooting is being executed in the imaging device; causes the output means to output the second three-dimensional shape information when continuous shooting is not being executed in the imaging device. The information processing apparatus according to claim 1, characterized by the above.

4. The state information includes information indicating the type of the subject being imaged by the imaging device, The control means: causes the output means to output the first three-dimensional shape information when the type of the subject being imaged by the imaging device is a predetermined subject type; causes the output means to output the second three-dimensional shape information when the type of the subject being imaged by the imaging device is not the predetermined subject type. The information processing apparatus according to claim 1, characterized by the above.

5. The predetermined subject type is a subject type in which the recommended shutter speed at the time of shooting is set shorter than a predetermined time, according to the information processing apparatus of claim 4.

6. The state information includes information on the shutter speed set in the imaging device, The control means: When the shutter speed set in the imaging device is shorter than a predetermined time, cause the first three-dimensional shape information to be output to the output means. When the shutter speed set in the imaging device is longer than the predetermined time, cause the second three-dimensional shape information to be output to the output means. The information processing apparatus according to claim 1, characterized in that.

7. The state information includes information indicating the shooting mode set in the imaging device. The shooting modes set in the imaging device are each associated with the required detection accuracy of the subject. The control means. When the shooting mode requiring the first detection accuracy is set in the imaging device, cause the first three-dimensional shape information to be output to the output means. When the shooting mode requiring the second detection accuracy higher than the first detection accuracy is set in the imaging device, cause the second three-dimensional shape information to be output to the output means. The information processing apparatus according to claim 1, characterized in that.

8. The information processing apparatus further includes a generation means for generating the second three-dimensional shape information based on the first three-dimensional shape information acquired by the first acquisition means. The first acquisition means acquires the second three-dimensional shape information generated by the generation means. The information processing apparatus according to claim 1, characterized in that.

9. The control means further controls the generation means not to generate the second three-dimensional shape information when causing the first three-dimensional shape information to be output to the output means. The information processing apparatus according to claim 8, characterized in that.

10. The information processing apparatus further includes a distance measuring means for performing distance measurement on the scene to generate the three-dimensional shape information. The first acquisition means acquires the three-dimensional shape information generated by the distance measuring means. The information processing apparatus according to claim 8, characterized in that.

11. The information processing apparatus further includes a distance measuring means for performing distance measurement on the scene to generate the three-dimensional shape information. The first acquisition means acquires the three-dimensional shape information generated by the distance measuring means. The control means. Controls the spatial resolution of the distance measurement by the distance measuring means according to the state information. Causes the three-dimensional shape information acquired by the first acquisition means to be output to the output means. The information processing apparatus according to claim 1, characterized in that.

12. A first acquisition step of acquiring three-dimensional shape information of a scene imaged by an imaging device. A second acquisition step of acquiring state information regarding the imaging state of the imaging device. An output step of outputting the three-dimensional shape information to the imaging device; A control step of controlling the operation in the output step; and having In the first acquisition step, for the scene, at least one of first three-dimensional shape information and second three-dimensional shape information having a higher spatial resolution than the first three-dimensional shape information is acquired; In the control step, which of the first three-dimensional shape information and the second three-dimensional shape information is to be output is controlled according to the state information. A control method for an information processing apparatus, characterized by the above.

13. A program for causing a computer to function as each means of the information processing apparatus according to any one of Claims 1 to 11.

14. An imaging system including an imaging device, a distance measuring device that measures distance for a scene imaged by the imaging device to generate three-dimensional shape information, and an information processing device that controls the output of the three-dimensional shape information to the imaging device, wherein the information processing device has a first acquisition means for acquiring the three-dimensional shape information from the distance measuring device; a second acquisition means for acquiring state information regarding the imaging state of the imaging device from the imaging device; an output means for outputting the three-dimensional shape information to the imaging device; a control means for controlling the operation of the output means; and having The first acquisition means acquires at least one of first three-dimensional shape information and second three-dimensional shape information having a higher spatial resolution than the first three-dimensional shape information for the scene; The control means controls which of the first three-dimensional shape information and the second three-dimensional shape information is to be output according to the state information. An imaging system, characterized by the above.

15. The imaging device has an imaging optical system; imaging means; acquisition means for acquiring the three-dimensional shape information output by the output means; determination means for determining a focus position based on the three-dimensional shape information acquired by the acquisition means; changing means for changing the state of the imaging optical system based on the focus position determined by the determination means; and having The imaging system according to Claim 14, characterized by the above.

16. The imaging device has imaging means; acquisition means for acquiring the three-dimensional shape information output by the output means; detection means for detecting a subject based on the three-dimensional shape information acquired by the acquisition means; exposure control means for performing exposure control of the imaging means based on the detection result by the detection means; and having The imaging system according to claim 14, characterized in that...

Images