Image sensing device, image sensing method, and program

Abstract

An image sensing device (100) is provided with: a 2D camera (10) that generates a two-dimensional image; a 3D camera (20) that has a light source (21) and generates a distance image on the basis of reflected light of light irradiated by the light source (21); an image recognition unit (30) that uses the two-dimensional image or the distance image to recognize a subject reflected in the two-dimensional image; and a camera control unit (40) that controls the on / off of the operation of the 3D camera (20), the camera control unit (40) turning on the operation of the 3D camera (20) on the basis of the result of recognition using the two-dimensional image when the 3D camera (20) is in a standby state in which the light source (21) does not emit light after the initialization process is completed after the 3D camera (20) is turned on, and turning off the operation of the 3D camera (20) on the basis of the result of recognition using the two-dimensional image. When the 3D camera (20) is operated by the camera control unit (40) and then the 3D camera (20) is operated by the camera control unit (40), the 3D camera (20) is in a standby state.

Description

Technical Field

[0001] The present disclosure relates to an image sensing device, an image sensing method, and a program. Background Art

[0002] For example, Patent Document 1 discloses a device for performing personal authentication or determining the presence or absence of a person. In this device, a 2D camera and a 3D camera are used in combination. Thereby, it is possible to prevent impersonation using a photograph or to detect only an object within a specified distance range.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: WO 2020 / 075525 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] However, since the 3D camera includes a light emitting device (light source) for measuring distance, in the case of using a 3D camera as in the device disclosed in Patent Document 1, power consumption increases along with the light emission of the light source.

[0008] Therefore, the present disclosure provides an image sensing device and the like that can reduce power consumption while using a 2D camera and a 3D camera in combination.

[0009] Means for Solving the Problems

[0010] The image sensing device according to the present disclosure includes: a 2D camera that generates a two-dimensional image; a 3D camera that has a light source and generates a distance image based on the reflected light of the light irradiated by the light source; an image recognition unit that uses the two-dimensional image or the distance image to perform recognition of a subject reflected in the two-dimensional image; and a camera control unit that controls the activation and deactivation of the operation of the 3D camera. The camera control unit activates the operation of the 3D camera based on the result of the recognition using the two-dimensional image when the initialization process after the 3D camera is activated is completed and the light source is not emitting light, and the 3D camera enters the standby state when the operation of the 3D camera is deactivated by the camera control unit after the operation is activated by the camera control unit.

[0011] The image sensing method according to the present disclosure is an image sensing method performed by an image sensing device, the image sensing device including: a 2D camera that generates a two-dimensional image; and a 3D camera that has a light source and generates a distance image based on the reflected light of the light irradiated by the light source. The image sensing method includes: an image recognition step of using the two-dimensional image or the distance image to perform recognition of a subject imaged in the two-dimensional image; and a camera control step of controlling the activation and deactivation of the operation of the 3D camera. In the camera control step, when the initialization process after the 3D camera is activated is completed and the light source is not emitting light, that is, in a standby state, based on the result of the recognition using the two-dimensional image, the operation of the 3D camera is activated. When the 3D camera is deactivated through the camera control step after the operation is activated through the camera control step, it becomes the standby state.

[0012] The program of the present disclosure is a program that causes a computer to execute the above-described image sensing method.

[0013] Advantages of the Invention

[0014] According to an image sensing device or the like according to an aspect of the present disclosure, it is possible to reduce power consumption while using a 2D camera and a 3D camera in combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a block diagram showing an example of the image sensing device according to Embodiment 1.

[0016] Figure 2 It is a timing chart showing a first example of the operation of the image sensing device according to Embodiment 1.

[0017] Figure 3 It is a timing chart showing a second example of the operation of the image sensing device according to Embodiment 1.

[0018] Figure 4 It is a timing chart showing a third example of the operation of the image sensing device according to Embodiment 1.

[0019] Figure 5 It is a timing chart showing a fourth example of the operation of the image sensing device according to Embodiment 1.

[0020] Figure 6 It is a block diagram showing an example of the image sensing device according to Embodiment 2.

[0021] Figure 7 It is a block diagram showing an example of the image sensing device according to Embodiment 3.

[0022] Figure 8 It is a flowchart showing an example of an image sensing method according to another embodiment. Detailed implementation manners

[0023] Hereinafter, the implementation manners will be specifically described with reference to the accompanying drawings.

[0024] In addition, the implementation manners described below are all general or specific examples. The numerical values, shapes, materials, constituent elements, arrangement positions of the constituent elements, connection manners, steps, order of steps, etc. shown in the following implementation manners are examples and are not intended to limit the present disclosure.

[0025] (Embodiment 1)

[0026] Hereinafter, Figures 1 to 5 the image sensing device of Embodiment 1 will be described.

[0027] Figure 1 is a block diagram showing an example of the image sensing device 100 of Embodiment 1. In addition, Figure 1 the MPU (Micro Processing Unit) 200 that processes the information output from the image sensing device 100 is shown. In addition, the MPU 200 may also be a constituent element of the image sensing device 100.

[0028] The image sensing device 100 is a device for recognizing a subject imaged in an image, and includes a 2D camera 10, a 3D camera 20, an image recognition unit 30, a camera control unit 40, a 2D output control unit 51, and a 3D output control unit 52. In addition, Figure 1 in Figure 1 the image recognition unit 30 is shown as a constituent element included in the camera control unit 40, but the image recognition unit 30 may not be a constituent element of the camera control unit 40. In addition,

[0029] the 2D output control unit 51 and the 3D output control unit 52 are collectively shown as an output control unit 50.

[0030] The 2D camera 10 is a camera that generates a two-dimensional image. The 2D camera 10 is, for example, a sensor such as an image sensor or an RGB sensor. In addition, the two-dimensional image generated by the 2D camera 10 may also be black and white.

[0031] For example, the 2D camera 10 includes a control unit 11. The control unit 11 is a processing unit that controls the focusing and exposure of the 2D camera 10. In addition, the control unit 11 may be provided in the image sensing device 100 and may not be provided in the 2D camera 10. For example, the control unit 11 controls the focusing and exposure of the 2D camera 10 according to a control signal from the camera control unit 40.

[0032] The 3D camera 20 is a camera that has a light source 21 and generates a distance image based on the reflected light of the light irradiated by the light source 21. When the 3D camera 20 obtains a signal for activating its own operation, it causes the light source 21 to emit light (e.g., intermittent light emission) at a predetermined timing, and generates a distance image according to the signal after exposure. In addition, when the 3D camera 20 obtains a signal for deactivating its own operation, it turns off the light emission of the light source 21 and stops generating the distance image. In this case, the light emission of the light source 21 remains off. In addition, the 3D camera 20 may also cause the light source 21 to emit light during the period when it obtains a signal for activating its own operation, and turn off the light emission of the light source 21 when it no longer obtains a signal for activating its own operation. The 3D camera 20 is, for example, a camera using the TOF (Time Of Flight) method, or LiDAR (Light Detection And Ranging), etc.

[0033] In order to generate a distance image, the 3D camera 20 requires the light emission of the light source 21, so the power consumption is large. On the other hand, the 2D camera 10 basically does not require a light source, so the power consumption is small.

[0034] The image recognition unit 30 uses the two-dimensional image generated by the 2D camera 10 or the distance image generated by the 3D camera 20 to recognize the subject reflected in the two-dimensional image. For example, in the recognition, it is possible to recognize whether a predetermined object is reflected in the two-dimensional image. The predetermined object is the recognition object and is appropriately set according to the use, etc. For example, when using the image sensing device 100 to recognize a person's face, the predetermined object becomes a person's face. For example, when a dog's face is reflected in the two-dimensional image, the recognition result indicates that a person's face is not reflected in the two-dimensional image (in addition, it may also indicate that a dog's face is reflected according to the performance of the image recognition unit 30), and when a person's face is reflected in the two-dimensional image, the recognition result indicates that a person's face is reflected in the two-dimensional image. Here, the predetermined object is not limited to a person's face.

[0035] The camera control unit 40 controls the activation and deactivation of the operation of the 3D camera 20. Specifically, when activating the operation of the 3D camera 20, the camera control unit 40 outputs a signal for activating the operation of the 3D camera 20 to the 3D camera 20, and when deactivating the operation of the 3D camera 20, the camera control unit 40 outputs a signal for deactivating the operation of the 3D camera 20 to the 3D camera 20. In addition, the camera control unit 40 may continuously output a signal for activating the operation of the 3D camera 20 to the 3D camera 20 when activating the operation of the 3D camera 20, and stop outputting the signal for activating the operation of the 3D camera 20 when deactivating the operation of the 3D camera 20. When the 3D camera 20 is in the standby state, the camera control unit 40 activates the operation of the 3D camera 20 based on the result of recognition using a two-dimensional image. The standby state of the 3D camera 20 is a state in which the initialization process after startup is completed and the light source 21 remains off. That is, the standby state of the 3D camera 20 is a state in which the power of the 3D camera 20 is turned on but the light source 21 remains off. When the operation of the 3D camera 20 is deactivated by the camera control unit 40 after the operation of the 3D camera 20 is activated, the 3D camera 20 enters the standby state. Thus, controlling the activation and deactivation of the operation of the 3D camera 20 does not mean controlling the power-on and power-off of the 3D camera 20.

[0036] For example, when the 3D camera 20 is in the standby state, if the result of recognition using a two-dimensional image indicates that a predetermined object is reflected in the two-dimensional image, the camera control unit 40 activates the operation of the 3D camera 20. In addition, when the operation of the 3D camera 20 is activated, the camera control unit 40 deactivates the operation of the 3D camera 20 based on the result of recognition using a two-dimensional image or a distance image. Specific examples of the operation of the camera control unit 40 will be described later.

[0037] The 2D output control unit 51 controls the output of the two-dimensional image based on the result of recognition using a two-dimensional image or a distance image. For example, the 2D output control unit 51 extracts a part of the two-dimensional image based on the recognition result and outputs it to the MPU 200.

[0038] The 3D output control unit 52 controls the output of the distance image based on the result of recognition using a two-dimensional image or a distance image. For example, the 3D output control unit 52 extracts a part of the distance image based on the recognition result and outputs it to the MPU 200.

[0039] The distance image of the 3D camera 20 may become very large data, so there are problems such as a decrease in the processing speed of the MPU 200 for analyzing two-dimensional images and distance images or performing authentication using them, and an increase in power consumption caused by an increase in the processing load. In response, for example, by controlling to output a part of the two-dimensional image and the distance image according to the recognition result, the processing load in the MPU 200 can be reduced. Therefore, the processing speed of the MPU 200 can be increased, and in addition, an increase in power consumption caused by an increase in the processing load can be suppressed.

[0040] The MPU 200 is the output destination of the two-dimensional image and the distance image, and is a processing unit that analyzes the two-dimensional image and the distance image or performs processing such as authentication using them. For example, when the subject reflected in the two-dimensional image is recognized as a human face, the MPU 200 recognizes the individual differences of the face to perform face authentication. In addition, face authentication is an example, and the object of authentication based on the MPU 200 is not particularly limited.

[0041] Next, use Figures 2 to 5 To illustrate a specific example of the operation of the image sensing device 100.

[0042] First, use Figure 2 To illustrate the first example of the operation of the image sensing device 100.

[0043] Figure 2 Is a timing chart showing the first example of the operation of the image sensing device 100 according to Embodiment 1.

[0044] First, the 2D camera 10 generates a surrounding two-dimensional image (step S101).

[0045] At this time, the 3D camera 20 is in a standby state, and the light source 21 is in a non-luminous state (step S102).

[0046] The 2D camera 10 outputs the generated two-dimensional image (step S103). For example, the 2D camera 10 outputs the two-dimensional image to the image recognition unit 30 and the 2D output control unit 51. In addition, the 2D output control unit 51 outputs the two-dimensional image to the MPU 200. In the first example, the 2D output control unit 51 outputs the two-dimensional image as it is to the MPU 200. For example, the MPU 200 outputs the two-dimensional image to a display or the like.

[0047] The image recognition unit 30 uses a two-dimensional image to recognize the subject imaged in the two-dimensional image. Specifically, the image recognition unit 30 performs face recognition to recognize whether a predetermined object (such as a human face) is imaged in the two-dimensional image (step S104). For example, when a person moves into the field of view angle of the 2D camera 10, the result of the recognition using the two-dimensional image indicates that a human face is imaged in the two-dimensional image. For example, the predetermined object to be recognized is set by the MPU 200.

[0048] When a face is detected in the two-dimensional image, the image recognition unit 30 notifies the camera control unit 40 that a face has been detected (step S105).

[0049] Upon receiving the notification that a face has been detected, the camera control unit 40 activates the operation of the 3D camera 20 (step S106). For example, the camera control unit 40 outputs a signal to the 3D camera 20 to activate the operation of the 3D camera 20. In this way, when the 3D camera 20 is in the standby state, if the result of the recognition using the two-dimensional image indicates that a human face is imaged in the two-dimensional image, the camera control unit 40 activates the operation of the 3D camera 20. That is, the camera control unit 40 does not activate the operation of the 3D camera 20 (in other words, does not turn on the light source 21) before the recognition object is imaged in the two-dimensional image, and activates the operation of the 3D camera 20 when the recognition object is imaged in the two-dimensional image, so that the 3D camera 20 can generate a distance image. Thereby, before the face of the person as the recognition object is imaged in the two-dimensional image, the 3D camera 20 can be put into the standby state (that is, keep the light source 21 turned off), and power consumption can be reduced.

[0050] The 3D camera 20 starts to emit light and perform exposure upon receiving the signal for activating the operation of the 3D camera 20, generates a surrounding distance image (step S107), and outputs it (step S108). For example, the 3D camera 20 outputs the distance image to the image recognition unit 30 and the 3D output control unit 52. In addition, the 3D output control unit 52 outputs the distance image to the MPU 200. In the first example, the 3D output control unit 52 outputs the distance image to the MPU 200 as it is. For example, the MPU 200 uses the two-dimensional image and the distance image for face authentication and the like.

[0051] The image recognition unit 30 uses the two-dimensional image or the distance image to recognize the subject imaged in the two-dimensional image. Specifically, the image recognition unit 30 performs face recognition to recognize whether a predetermined object (such as a human face) is no longer imaged in the two-dimensional image (step S109). For example, when a person moves away from the field of view angle of the 2D camera 10, the result of the recognition using the two-dimensional image indicates that a human face is not imaged in the two-dimensional image.

[0052] In the case where no face is detected in the two-dimensional image, the image recognition unit 30 notifies the camera control unit 40 that no face is detected any longer (step S110).

[0053] Upon receiving the notification that no face is detected any longer, the camera control unit 40 shuts down the operation of the 3D camera 20 (step S111). For example, the camera control unit 40 outputs a signal for shutting down the operation of the 3D camera 20 to the 3D camera 20. In this way, when the 3D camera 20 is operating, the camera control unit 40 shuts down the operation of the 3D camera 20 based on the result of recognition using the two-dimensional image or the distance image.

[0054] Then, the 3D camera 20 returns to the standby state again (step S112). In this way, after the operation of the 3D camera 20 is turned on, when the face of the person to be recognized no longer appears in the two-dimensional image, since there is no longer an object to be recognized, the operation of the 3D camera 20 can be shut down (that is, the 3D camera 20 can be made to enter the standby state). Along with this, the light emission of the light source 21 also remains turned off in the standby state, and the power consumption of the 3D camera 20 can be reduced.

[0055] Next, Figure 3 A second example of the operation of the image sensing device 100 will be described.

[0056] Figure 3 FIG. is a timing chart showing a second example of the operation of the image sensing device 100 according to Embodiment 1.

[0057] Figure 3 The processing of steps S201 to S207 shown in Figure 2 is the same as the processing of steps S101 to S107 shown in

[0058] and thus the description thereof is omitted.

[0059] The 3D camera 20 outputs the generated distance image (step S208). For example, the 3D camera 20 outputs the distance image to the image recognition unit 30 and the 3D output control unit 52.

[0060] The image recognition unit 30 uses the two-dimensional image and the distance image to perform recognition of the subject imaged in the two-dimensional image. Specifically, the image recognition unit 30 performs face recognition to determine whether a pre-determined object (such as a human face) imaged in the two-dimensional image is actually present in the image (step S210). For example, when an image (such as a photo) showing a human face is within the field of view of the 2D camera 10, the result of the recognition using the two-dimensional image and the distance image indicates that the human face imaged in the two-dimensional image is the same as the one in the photo. Since the photo is flat while the actual human face has depth, by using the distance image, it is possible to identify whether the human face imaged in the two-dimensional image is a real human face or just a face in a photo.

[0061] When a photo is detected in the two-dimensional image, the image recognition unit 30 notifies the camera control unit 40 that a photo has been detected (step S211).

[0062] Upon receiving the notification of photo detection, the camera control unit 40 outputs a stop signal to the 3D output control unit 52 (step S212). As a result, the 3D output control unit 52 can discard the distance image obtained in step S208 and not output the distance image to the MPU 200, thereby reducing the processing load on the MPU 200.

[0063] In addition, upon receiving the notification of photo detection, the camera control unit 40 turns off the operation of the 3D camera 20 (step S213). For example, the camera control unit 40 outputs a signal to the 3D camera 20 to turn off its operation. In this way, when the operation of the 3D camera 20 is turned on and the result of the recognition using the two-dimensional image and the distance image indicates that the human face imaged in the two-dimensional image is the same as the one in the photo, the camera control unit 40 turns off the operation of the 3D camera 20.

[0064] Then, the 3D camera 20 returns to the standby state again (step S214). In this way, after the operation of the 3D camera 20 is turned on, if the human face recognized through the distance image is a face in a photo, since the recognized human face is not a real object and not the object to be recognized, the operation of the 3D camera 20 can be turned off (i.e., the 3D camera 20 can be put into the standby state). Along with this, the light emission of the light source 21 also remains off in the standby state, reducing the power consumption of the 3D camera 20.

[0065] In addition, when no photo is detected, the processing after step S109 described in the first example is performed.

[0066] Next, Figure 4 A third example of the operation of the image sensing device 100 will be described.

[0067] Figure 4 1 is a timing chart showing a third example of the operation of the image sensing device 100 according to the first embodiment.

[0068] Figure 4 The processing of steps S301 to S306 shown in FIG. Figure 2 The processing of steps S101 to S106 shown is the same, so the description is omitted. In addition, in the first example, it is described that the 2D output control unit 51 outputs the two-dimensional image as it is to the MPU 200, but in the third example, the 2D output control unit 51 controls the output of the two-dimensional image based on the result of the recognition using the two-dimensional image or the distance image.

[0069] When a predetermined object (e.g., a human face) is shown in the two-dimensional image, the image recognition unit 30 outputs position information of a portion of the two-dimensional image where the human face is shown to the 2D output control unit 51 and the 3D output control unit 52 (step S307). For example, the image recognition unit 30 outputs information indicating pixels corresponding to a region in the two-dimensional image where the human face is detected as position information to the 2D output control unit 51 and the 3D output control unit 52.

[0070] The 3D camera 20 starts to emit light and expose by receiving a signal for starting the operation of the 3D camera 20, generates a distance image of the surroundings (step S308), and outputs it (step S309). For example, the 3D camera 20 outputs the distance image to the image recognition unit 30 and the 3D output control unit 52. In addition, in the first example, it is described that the 3D output control unit 52 outputs the distance image as it is to the MPU 200, but in the third example, the 3D output control unit 52 controls the output of the distance image based on the result of the recognition using the two-dimensional image or the distance image.

[0071] Based on the acquired position information, the 2D output control unit 51 outputs the 2D image of the portion of the 2D image where the face of the person is reflected to the MPU 200 (step S310). For example, the 2D output control unit 51 extracts the pixel portion corresponding to the region detected as the face of the person in the 2D image from the 2D image, and outputs the pixel portion in the 2D image to the MPU 200.

[0072] In addition, based on the acquired position information, the 3D output control unit 52 outputs the distance image of the portion of the two-dimensional image where the face of the person is reflected to the MPU 200 (step S311). For example, the 3D output control unit 52 extracts the pixel portion corresponding to the region detected as the face of the person in the two-dimensional image from the distance image, and outputs the pixel portion in the distance image to the MPU 200.

[0073] In this way, when the result of recognition using a two-dimensional image or a distance image indicates that a predetermined object (e.g., a person's face) is reflected in the two-dimensional image, the 2D output control unit 51 outputs a two-dimensional image of a portion of the two-dimensional image where the person's face is reflected, and the 3D output control unit 52 outputs a distance image of a portion of the two-dimensional image where the person's face is reflected. When the face of a person to be recognized is reflected in the two-dimensional image, only the two-dimensional image of the portion where the person's face is reflected (in other words, a two-dimensional image where a portion where the person's face is not reflected is removed) can be output, and also, only the distance image of the portion where the person's face is reflected (in other words, a distance image where a portion where the person's face is not reflected is removed) can be output. Therefore, the processing load in the MPU 200 can be reduced.

[0074] In addition, in the case where multiple faces are detected in step S304, or the face of a person shown in the two-dimensional image is located far away, it is sometimes desired to authenticate a face located within a certain distance. In such a case, the image recognition unit 30 outputs information on pixels corresponding to an area within a specified distance range in an area in the two-dimensional image where a person's face is detected to the 2D output control unit 51 and the 3D output control unit 52. Thus, in step S310, the 2D output control unit 51 can output a two-dimensional image of a portion of the two-dimensional image where a person's face is shown within a specified distance range to the MPU 200. In addition, in step S311, the 3D output control unit 52 can output a distance image of a portion of the two-dimensional image where a person's face is shown within a specified distance range to the MPU 200.

[0075] In this way, when the result of recognition using the two-dimensional image and the distance image indicates that a person's face is reflected within a predetermined distance range, the 2D output control unit 51 may output a two-dimensional image of a portion of the two-dimensional image where the person's face is reflected, and the 3D output control unit 52 may output a distance image of a portion of the two-dimensional image where the person's face is reflected. When a face of a person to be recognized is reflected in the two-dimensional image and the face of the person is within a predetermined distance range, only the two-dimensional image of the portion may be output, and also only the distance image of the portion may be output. Therefore, the processing load in the MPU 200 may be reduced.

[0076] In addition, although Figure 4 Although illustration is omitted, in the third example, the processing after step S109 is performed in the same manner as in the first example, or the processing after step S209 is performed in the same manner as in the second example.

[0077] Next, use Figure 5 A fourth example of the operation of the image sensing device 100 will be described.

[0078] Figure 5 This is a timing chart showing a fourth example of the operation of the image sensing device 100 according to Embodiment 1.

[0079] Figure 5 The processing of steps S401 to S408 shown is Figure 2 the same as the processing of steps S101 to S108 shown, and thus the description thereof is omitted.

[0080] The image recognition unit 30 recognizes the distance to the face of a person reflected in the two-dimensional image based on the distance image, and outputs the distance information to the camera control unit 40 (step S409). The camera control unit 40 outputs the distance information to the 2D camera 10 (specifically, the control unit 11) (step S410).

[0081] The 2D camera 10 (control unit 11) controls the focus of the 2D camera 10 according to the acquired distance information (step S411), and outputs the two-dimensional image acquired by the 2D camera 10 whose focus has been controlled (step S412).

[0082] In this way, the control unit 11 controls the focus of the 2D camera 10 based on the distance information indicating the distance to the face of a person reflected in the two-dimensional image. According to the distance from the image sensing device 100 to the face of a person reflected in the two-dimensional image, the focus of the 2D camera 10 can be controlled at high speed. Therefore, it is easy to focus on the face of a person reflected in the two-dimensional image, and the accuracy of recognition or authentication using the two-dimensional image can be improved.

[0083] In addition, the control unit 11 may also control the exposure of the 2D camera 10 based on the position information of the face of a person reflected in the two-dimensional image.

[0084] In addition, although Figure 5 the illustration is omitted, in the fourth example, the processing after step S109 is also performed in the same manner as in the first example, or the processing after step S209 is performed in the same manner as in the second example.

[0085] As described above, the 3D camera 20 can be set to the standby state until the two-dimensional image generated by the 2D camera 10 obtains the result of the target recognition (for example, detects a human face). The 3D camera 20 in the standby state is activated (in other words, the power is turned on), but the light source 21 does not emit light, and only the power consumption at the standby power level is generated. Therefore, the power consumption of the 3D camera 20 is reduced. Therefore, before the result of the target recognition is obtained from the two-dimensional image, the power consumption can be reduced without causing the light source 21 of the 3D camera 20 to emit light. When the result of the target recognition is obtained, the light source 21 can be made to emit light for detailed recognition by starting the operation of the 3D camera 20. After that, when the operation of the 3D camera 20 is turned off, the 3D camera 20 becomes the standby state again, and the power consumption of the 3D camera 20 can be reduced. Therefore, it is possible to reduce the power consumption while using the 2D camera 10 and the 3D camera 20 in combination. In addition, in the image sensing device 100, the operation of the 3D camera 20 is controlled to be turned on and off based on the recognition result, but the power supply of the 3D camera 20 is not controlled based on the recognition result. Therefore, it is possible to suppress the initialization process from being performed every time the power supply of the 3D camera 20 is repeatedly turned on and off, and it is possible to suppress the delay.

[0086] (Embodiment 2)

[0087] Next, Figure 6 the image sensing device of Embodiment 2 will be described.

[0088] Figure 6 is a block diagram showing an example of the image sensing device 100a of Embodiment 2. In addition, Figure 6 shows the MPU 200 that processes the information output from the image sensing device 100a. In addition, the MPU 200 may also be a component of the image sensing device 100a.

[0089] The difference between the image sensing device 100a and the image sensing device 100 in Embodiment 1 is that it includes a 2D camera 10a instead of the 2D camera 10, includes a plurality of 3D cameras instead of the 3D camera 20, and includes a camera control unit 40a instead of the camera control unit 40. Other aspects are the same as those in Embodiment 1, so the description is omitted.

[0090] Since the image sensing device 100a includes a plurality of 3D cameras, it is possible to perform a wider range of sensing compared to the case of including one 3D camera. For example, the image sensing device 100a can be used for obstacle detection processing that requires a wide range of sensing.

[0091] As Figure 6As shown, the image sensing device 100a includes, for example, 3D cameras 20a and 20b as multiple 3D cameras. The 3D camera 20a has a light source 21a and generates a distance image based on the reflected light of the light irradiated by the light source 21a. The 3D camera 20b has a light source 21b and generates a distance image based on the reflected light of the light irradiated by the light source 21b. Additionally, the image sensing device 100a may include three or more 3D cameras.

[0092] The 2D camera 10a is a camera that generates a two-dimensional image. The 2D camera 10a is, for example, a wide field-of-view camera. Since the image sensing device 100a includes multiple 3D cameras, accordingly, the 2D camera 10a also becomes a wide field-of-view camera.

[0093] In addition to the functions of the camera control unit 40 in Embodiment 1, the camera control unit 40a has the following functions: when multiple 3D cameras are in a standby state, based on the position of the subject reflected in the two-dimensional image, at least one 3D camera is determined from the multiple 3D cameras, and based on the result of the recognition using the two-dimensional image, the operation of the determined at least one 3D camera is turned on. For example, when the subject reflected in the two-dimensional image is recognized by the image recognition unit 30 to be within the field of view of the 3D camera 20a, the camera control unit 40a determines the 3D camera 20a from the 3D cameras 20a and 20b. Then, the camera control unit 40a, for example, turns on the operation of the determined 3D camera 20a when the subject reflected in the two-dimensional image is a pre-determined object (such as a human face, etc.).

[0094] In this way, by only turning on the operation of the 3D camera 20a that can generate the distance image of the position of the subject reflected in the two-dimensional image among the 3D cameras 20a and 20b, compared with the case of turning on the operations of all the 3D cameras 20a and 20b, power consumption can be reduced. Additionally, since the distance images from all the 3D cameras 20a and 20b do not need to be output to the MPU 200, the processing load in the MPU 200 can be reduced. Therefore, the processing speed of the MPU 200 can be increased, and additionally, the increase in power consumption caused by the increase in processing load can be suppressed.

[0095] Additionally, regarding the image sensing device 100a in Embodiment 2, it may also have the functions of the image sensing device 100 described in the first to fourth examples of Embodiment 1.

[0096] (Embodiment 3)

[0097] Next, Figure 7 the image sensing device of Embodiment 3 will be described.

[0098] Figure 7This is a block diagram showing an example of the image sensing device 100b according to Embodiment 3. Additionally, in Figure 7 the MPU 200 that processes the information output from the image sensing device 100b is shown. Additionally, the MPU 200 may also be a component of the image sensing device 100b.

[0099] The difference between the image sensing device 100b and the image sensing device 100 in Embodiment 1 is that it includes a BW-TOF camera 110 instead of the 2D camera 10 and the 3D camera 20. Other aspects are the same as those in Embodiment 1, so the description thereof is omitted.

[0100] The BW-TOF camera 110 is a camera in which a 2D camera and a 3D camera are integrally provided. In other words, the BW-TOF camera 110 is a camera having both the functions of a 2D camera and a 3D camera.

[0101] The BW-TOF camera 110 includes a BW-TOF sensor 111 and a light source 112. The BW-TOF sensor 111 is configured with pixels (IR pixels) for generating a distance image and pixels (BW pixels) for generating a two-dimensional image (black-and-white image). When the light source 112 is not emitting light, it can generate a two-dimensional image, and based on the reflected light of the light irradiated by the light source 112, it can generate both a distance image and a two-dimensional image.

[0102] In addition, here, the BW-TOF camera 110 is shown as an example of a camera in which a 2D camera and a 3D camera are integrally provided. However, as long as it is a camera in which a 2D camera and a 3D camera are integrally provided, in other words, as long as it is a camera having both the functions of a 2D camera and a 3D camera, there is no particular limitation. For example, the camera in which a 2D camera and a 3D camera are integrally provided may also be an RGB-TOF camera. The RGB-TOF camera is a camera that can acquire a color image instead of a black-and-white image with respect to the BW-TOF camera 110.

[0103] In this way, when the 2D camera and the 3D camera are provided integrally, it is also possible to reduce power consumption while using the 2D camera and the 3D camera simultaneously. Additionally, by providing the 2D camera and the 3D camera integrally, miniaturization of the image sensing device 100b can be achieved.

[0104] Additionally, regarding the image sensing device 100b in Embodiment 3, it may also have the functions of the image sensing device 100 described in the first to fourth examples of Embodiment 1.

[0105] (Other Embodiments)

[0106] As described above, the image sensing device according to one or more aspects of the present disclosure has been described based on the embodiments, but the present disclosure is not limited to these embodiments. As long as it does not deviate from the gist of the present disclosure, aspects obtained by applying various modifications conceivable by those skilled in the art to each embodiment, or aspects constructed by combining components in different embodiments, are included within the scope of one or more aspects of the present disclosure.

[0107] For example, in the above embodiment, an example was described in which the camera control unit turns off the operation of the 3D camera based on the result of recognition using a two-dimensional image or a distance image when the operation of the 3D camera is started, but it is not limited thereto. For example, the camera control unit may also turn off the operation of the 3D camera based on the input result from an external device when the operation of the 3D camera is started.

[0108] For example, the external device is a device that uses the result of recognition using a two-dimensional image or a distance image to authenticate a subject imaged in the two-dimensional image, specifically, an MPU or the like. When the authentication process is completed, the MPU outputs a signal indicating that meaning to the image sensing device. In this case, when the input result from the MPU to the image sensing device indicates that the authentication process has been completed, there is no longer a need for recognition of the subject imaged in the two-dimensional image, so the operation of the 3D camera can be turned off. Along with this, the light emission of the light source also remains off, and the power consumption of the 3D camera can be reduced.

[0109] For example, in the above embodiment, an example was described in which the image sensing device includes a 2D output control unit that controls the output of a two-dimensional image and a 3D output control unit that controls the output of a distance image, but the image sensing device may not include the 2D output control unit and the 3D output control unit.

[0110] For example, in the above embodiment, an example was described in which the image sensing device includes a control unit that controls the focusing of the 2D camera, but the image sensing device may not include such a control unit.

[0111] For example, the present disclosure can be implemented not only as an image sensing device but also as an image sensing method including steps (processes) performed by components constituting the image sensing device.

[0112] Figure 8 It is a flowchart showing an example of an image sensing method according to another embodiment.

[0113] The image sensing method is a method executed by an image sensing device, and the image sensing device includes: a 2D camera that generates a two-dimensional image; and a 3D camera that has a light source and generates a distance image based on the reflected light of the light irradiated by the light source, as Figure 8As shown, the image sensing method includes: an image recognition step (step S11) of recognizing a subject reflected in a two-dimensional image using the two-dimensional image or a distance image; and a camera control step of controlling the activation and deactivation of the operation of a 3D camera. In the camera control step, when the 3D camera is in a standby state where the initialization process after startup is completed and the light source is not emitting light, based on the result of recognition using the two-dimensional image, the operation of the 3D camera is activated (step S12). When the 3D camera is deactivated in the camera control step after being activated in the camera control step, it becomes the standby state.

[0114] For example, the present disclosure can be implemented as a program for causing a computer (processor) to execute the steps included in the image sensing method. Further, the present disclosure can be implemented as a non-volatile computer-readable recording medium such as a CD-ROM on which the program is recorded.

[0115] For example, when the present disclosure is implemented by a program (software), each step is executed by using hardware resources such as a CPU, a memory, and an input / output circuit of a computer to execute the program. That is, the CPU obtains data from the memory or the input / output circuit or the like and performs arithmetic operations, or outputs the arithmetic result to the memory or the input / output circuit or the like, thereby executing each step.

[0116] In addition, in the above-described embodiment, each component included in the image sensing device may be constituted by dedicated hardware, or may be implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0117] Part or all of the functions of the image sensing device according to the above-described embodiment are typically implemented by an LSI as an integrated circuit. They may be individually monolithicized or may be monolithicized in a manner including part or all of them. In addition, the integration into an integrated circuit is not limited to an LSI, and may also be implemented by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconstruct the connection and setting of circuit units inside the LSI may also be used.

[0118] (Supplementary Note)

[0119] Through the description of the above embodiments, the following technology is disclosed.

[0120] (Technology 1) An image sensing device includes: a 2D camera that generates a two-dimensional image; a 3D camera having a light source that generates a distance image based on reflected light of light irradiated by the light source; an image recognition unit that uses the two-dimensional image or the distance image to perform recognition of a subject imaged in the two-dimensional image; and a camera control unit that controls turning on and off of the operation of the 3D camera. When the initialization process after the 3D camera is started is completed and the light source is not emitting light, the camera control unit turns on the operation of the 3D camera based on the result of the recognition using the two-dimensional image. When the operation of the 3D camera is turned off by the camera control unit after the operation of the 3D camera is turned on by the camera control unit, the 3D camera enters the standby state.

[0121] Thus, the 3D camera can be in the standby state until the result of the recognition of the target is obtained from the two-dimensional image generated by the 2D camera. Although the 3D camera in the standby state is started (in other words, the power is turned on), the light source does not emit light, and only power consumption at the standby power level is generated. Therefore, the power consumption of the 3D camera is reduced. Therefore, before the result of the recognition of the target is obtained from the two-dimensional image, the power consumption can be reduced without making the light source of the 3D camera emit light. When the result of the recognition of the target is obtained, the light source of the 3D camera can be made to emit light by turning on the operation of the 3D camera to perform detailed recognition. After that, when the operation of the 3D camera is turned off, the 3D camera enters the standby state again, and the power consumption of the 3D camera can be reduced. Therefore, power consumption can be reduced while using the 2D camera and the 3D camera together. In addition, in the image sensing device, the turning on and off of the operation of the 3D camera is controlled based on the result of the recognition, but the turning on and off of the power of the 3D camera is not controlled based on the result of the recognition. Therefore, it is possible to suppress the initialization process from being performed every time the power of the 3D camera is turned on and off, and it is possible to suppress delay.

[0122] (Technology 2) The image sensing device according to Technology 1, wherein when the 3D camera is in the standby state, the camera control unit turns on the operation of the 3D camera when the result of the recognition using the two-dimensional image indicates that a predetermined object is imaged in the two-dimensional image.

[0123] Thus, the 3D camera can be in the standby state until a predetermined object to be recognized is imaged in the two-dimensional image, and power consumption can be reduced.

[0124] (Technology 3) The image sensing device according to Technology 1 or 2, wherein when the 3D camera is turned on, the camera control unit turns off the operation of the 3D camera based on the result of the recognition using the two-dimensional image or the distance image.

[0125] Thus, based on the result of a certain recognition using a two-dimensional image or a distance image, the operation of the 3D camera can be turned off. Along with this, the light emission of the light source also remains off, and the power consumption of the 3D camera can be reduced.

[0126] (Technique 4) The image sensing device according to Technique 3, wherein when the 3D camera is turned on, if the result of the recognition using the two-dimensional image or the distance image indicates that a predetermined object is not reflected in the two-dimensional image, the operation of the 3D camera is turned off.

[0127] For example, after the operation of the 3D camera is turned on, if a predetermined object to be recognized no longer appears in the two-dimensional image, since there is no recognition object, the operation of the 3D camera can be turned off. Along with this, the light emission of the light source also remains off, and the power consumption of the 3D camera can be reduced.

[0128] (Technique 5) The image sensing device according to Technique 3 or 4, wherein when the 3D camera is turned on, if the result of the recognition using the two-dimensional image and the distance image indicates that a predetermined object reflected in the image is reflected in the two-dimensional image, the operation of the 3D camera is turned off.

[0129] For example, after the operation of the 3D camera is turned on, if the object recognized through the distance image is the object reflected in the image, since the recognized object is not a real object and is not the object to be recognized, the operation of the 3D camera can be turned off. Along with this, the light emission of the light source also remains off, and the power consumption of the 3D camera can be reduced.

[0130] (Technique 6) The image sensing device according to any one of Techniques 1 to 5, wherein when the 3D camera is turned on, based on the input result from an external device, the operation of the 3D camera is turned off.

[0131] For example, the external device is a device that uses the result of recognition using a two-dimensional image or a distance image to perform authentication on the subject reflected in the two-dimensional image, and outputs a signal indicating this meaning to the image sensing device when the authentication process is completed. In this case, when the input result from the external device to the image sensing device indicates that the authentication process has been completed, there is no longer a need for recognition of the subject reflected in the two-dimensional image, so the operation of the 3D camera can be turned off. Along with this, the light emission of the light source also remains off, and the power consumption of the 3D camera can be reduced.

[0132] (Technique 7) The image sensing device according to any one of Techniques 1 to 6, wherein the image sensing device further includes: a 2D output control unit that controls the output of the two-dimensional image based on the result of the recognition using the two-dimensional image or the distance image; and a 3D output control unit that controls the output of the distance image based on the result of the recognition using the two-dimensional image or the distance image.

[0133] The distance image of the 3D camera may become very large data, so there are problems such as a decrease in the processing speed of the device that is the output destination of the two-dimensional image and the distance image for analyzing the two-dimensional image and the distance image or performing authentication using them, and an increase in power consumption caused by an increase in the processing load. In response to this, for example, by controlling to output a part of the two-dimensional image and the distance image according to the recognition result, the processing load in the output destination device can be reduced. Therefore, the processing speed of the output destination device can be increased, and in addition, the increase in power consumption caused by an increase in the processing load can be suppressed.

[0134] (Technique 8) The image sensing device according to Technique 7, wherein when the result of the recognition using the two-dimensional image or the distance image indicates that a predetermined object is reflected in the two-dimensional image, the 2D output control unit outputs a two-dimensional image of the part in the two-dimensional image where the predetermined object is reflected, and the 3D output control unit outputs a distance image of the part in the two-dimensional image where the predetermined object is reflected.

[0135] When a predetermined object as an object to be recognized is reflected in the two-dimensional image, it is possible to output only the two-dimensional image of the part where the object is reflected (in other words, the two-dimensional image from which the part where the object is not reflected is removed), and in addition, it is possible to output only the distance image of the part where the object is reflected (in other words, the distance image from which the part where the object is not reflected is removed). Therefore, the processing load in the output destination device can be reduced.

[0136] (Technique 9) The image sensing device according to Technique 7, wherein when the result of the recognition using the two-dimensional image and the distance image indicates that a predetermined object is reflected within a predetermined distance range, the 2D output control unit outputs a two-dimensional image of the part in the two-dimensional image where the predetermined object is reflected, and the 3D output control unit outputs a distance image of the part in the two-dimensional image where the predetermined object is reflected.

[0137] When a predetermined object to be recognized is projected in a two-dimensional image and the object is within a specified distance range, it is possible to output only the part of the two-dimensional image in which the object is projected, and it is also possible to output only the distance image of the part in which the object is projected. Therefore, the processing load on the output destination device can be reduced.

[0138] (Technology 10) The image sensing device according to any one of Technologies 1 to 9, wherein the image sensing device further includes: a control unit that controls the focusing of the 2D camera based on the distance information from the image sensing device to the subject projected in the two-dimensional image represented by the distance image.

[0139] Thereby, it is possible to control the focusing of the 2D camera at high speed according to the distance from the image sensing device to the subject projected in the two-dimensional image. Therefore, it is easy to focus on the subject projected in the two-dimensional image, and the accuracy of recognition or authentication using the two-dimensional image can be improved.

[0140] (Technology 11) The image sensing device according to any one of Technologies 1 to 10, wherein the image sensing device includes a plurality of the 3D cameras, and when the plurality of 3D cameras are in the standby state, the camera control unit determines at least one of the 3D cameras from the plurality of 3D cameras based on the position of the subject projected in the two-dimensional image, and activates the determined at least one 3D camera based on the result of the recognition using the two-dimensional image.

[0141] For example, the image sensing device may be a device capable of performing wide-range sensing with a plurality of 3D cameras. In this case, by activating only the 3D camera that can generate the distance image of the position of the subject projected in the two-dimensional image, power consumption can be reduced compared to the case where the operations of all the plurality of 3D cameras are activated. In addition, it is not necessary to output the distance images from all the plurality of 3D cameras to the output destination device. Therefore, the processing load on the output destination device can be reduced. As a result, the processing speed of the output destination device can be increased, and in addition, an increase in power consumption caused by an increase in the processing load can be suppressed.

[0142] (Technology 12) The image sensing device according to any one of Technologies 1 to 11, wherein the 2D camera and the 3D camera are integrally provided.

[0143] In this way, by integrally providing the 2D camera and the 3D camera, miniaturization of the image sensing device can be achieved.

[0144] (Technology 13) An image sensing method, which is executed by an image sensing device. The image sensing device includes: a 2D camera that generates a two-dimensional image; and a 3D camera that has a light source and generates a distance image based on the reflected light of the light irradiated by the light source. The image sensing method includes: an image recognition step of recognizing a subject reflected in the two-dimensional image by using the two-dimensional image or the distance image; and a camera control step of controlling the turning on and off of the operation of the 3D camera. In the camera control step, when the initialization process after the 3D camera is started is completed and the light source is not emitting light, the operation of the 3D camera is turned on based on the result of the recognition using the two-dimensional image. When the operation of the 3D camera is turned off through the camera control step after the operation of the 3D camera is turned on through the camera control step, the 3D camera enters the standby state.

[0145] Accordingly, it is possible to provide an image sensing method capable of reducing power consumption while using a 2D camera and a 3D camera simultaneously.

[0146] (Technology 14) A program for causing a computer to execute the image sensing method according to Technology 13.

[0147] Accordingly, it is possible to provide a program capable of reducing power consumption while using a 2D camera and a 3D camera simultaneously.

[0148] Industrial Applicability

[0149] The present disclosure can be applied to devices that perform image recognition using a 2D camera and a 3D camera in combination, etc.

[0150] Description of Reference Numerals:

[0151] 10, 10a 2D camera

[0152] 11 Control unit

[0153] 20, 20a, 20b 3D camera

[0154] 21, 21a, 21b, 112 Light source

[0155] 30 Image recognition unit

[0156] 40, 40a Camera control unit

[0157] 50 Output control unit

[0158] 51 2D output control unit

[0159] 52 3D output control unit

[0160] 100, 100a, 100b Image sensing device

[0161] 110 BW-TOF Camera

[0162] 111 BW-TOF Sensor

[0163] 200 MPU

Claims

1. An image sensing device, comprising: A 2D camera that generates a two-dimensional image; A 3D camera having a light source that generates a distance image based on the reflected light of the light irradiated by the light source; An image recognition unit that uses the two-dimensional image or the distance image to recognize a subject imaged in the two-dimensional image; and A camera control unit that controls the activation and deactivation of the operation of the 3D camera, When the 3D camera is in a standby state where the initialization process after startup is completed and the light source does not emit light, the camera control unit activates the operation of the 3D camera based on the result of the recognition using the two-dimensional image, When the operation of the 3D camera is activated by the camera control unit and then deactivated by the camera control unit, the 3D camera enters the standby state.

2. The image sensing device according to claim 1, wherein, When the 3D camera is in the standby state, if the result of the recognition using the two-dimensional image indicates that a predetermined object is imaged in the two-dimensional image, the camera control unit activates the operation of the 3D camera.

3. The image sensing device according to claim 1 or 2, wherein, When the operation of the 3D camera is activated, the camera control unit deactivates the operation of the 3D camera based on the result of the recognition using the two-dimensional image or the distance image.

4. The image sensing device according to claim 3, wherein, When the operation of the 3D camera is activated, if the result of the recognition using the two-dimensional image or the distance image indicates that a predetermined object is not imaged in the two-dimensional image, the camera control unit deactivates the operation of the 3D camera.

5. The image sensing device according to claim 3 or 4, wherein, When the operation of the 3D camera is activated, if the result of the recognition using the two-dimensional image and the distance image indicates that a predetermined object imaged in the two-dimensional image is present, the camera control unit deactivates the operation of the 3D camera.

6. The image sensing device according to any one of claims 1 to 5, wherein, When the operation of the 3D camera is activated, the camera control unit deactivates the operation of the 3D camera based on the input result from an external device.

7. The image sensing device according to any one of claims 1 to 6, wherein, The image sensing device further comprises: A 2D output control unit that controls the output of the two-dimensional image based on the result of the recognition using the two-dimensional image or the distance image; and A 3D output control unit that controls the output of the distance image based on the result of the recognition using the two-dimensional image or the distance image.

8. The image sensing device according to claim 7, wherein, If the result of the recognition using the two-dimensional image or the distance image indicates that a predetermined object is imaged in the two-dimensional image, The 2D output control unit outputs a two-dimensional image of the part of the two-dimensional image in which the predetermined object is imaged, The 3D output control unit outputs a distance image of a portion of the two-dimensional image in which the predetermined object is reflected.

9. The image sensing device according to claim 7, wherein when the result of the recognition using the two-dimensional image and the distance image indicates that a predetermined object is reflected within a predetermined distance range, the 2D output control unit outputs a two-dimensional image of a portion of the two-dimensional image in which the predetermined object is reflected, and the 3D output control unit outputs a distance image of a portion of the two-dimensional image in which the predetermined object is reflected.

10. The image sensing device according to any one of claims 1 to 9, wherein the image sensing device further includes: a control unit that controls the focus of the 2D camera based on the distance information of the subject reflected in the two-dimensional image represented by the distance image.

11. The image sensing device according to any one of claims 1 to 10, wherein the image sensing device includes a plurality of the 3D cameras, and when the plurality of 3D cameras are in the standby state, the camera control unit determines at least one of the 3D cameras based on the position of the subject reflected in the two-dimensional image, and turns on the operation of the determined at least one 3D camera based on the result of the recognition using the two-dimensional image.

12. The image sensing device according to any one of claims 1 to 11, wherein the 2D camera is provided integrally with the 3D camera.

13. An image sensing method is an image sensing method executed by an image sensing device, wherein the image sensing device includes: 2D camera, generating a two-dimensional image; and a 3D camera having a light source and generating a distance image based on the reflected light of the light irradiated by the light source, the image sensing method includes: an image recognition step of recognizing a subject reflected in the two-dimensional image using the two-dimensional image or the distance image; and a camera control step of controlling the turning on and off of the operation of the 3D camera, wherein in the camera control step, when the 3D camera is in the standby state after the initialization process after startup is completed and the light source is not emitting light, the operation of the 3D camera is turned on based on the result of the recognition using the two-dimensional image, and when the operation of the 3D camera is turned off by the camera control step after the operation is turned on by the camera control step, the 3D camera becomes the standby state.

14. A program that causes a computer to execute the image sensing method according to claim 13.

Images