Image sensing device, image sensing method and program

Abstract

This image sensing device (100) comprises: a 2D camera (10) that generates a two-dimensional image; a 3D camera (20) that has a light source (21) and generates a distance image on the basis of the reflected light of a light irradiated by the light source (21); an image recognition unit (30) that uses the two-dimensional image or the distance image to perform a recognition of a subject appearing in the two-dimensional image; and a camera control unit (40) that controls the on and off of the operation of the 3D camera (20). When the 3D camera (20) is in a standby state in which the light source (21) is emitting no light after the completion of an initialization process following an activation, the camera control unit (40) turns on the operation of the 3D camera (20) on the basis of the result of the recognition using the two-dimensional image. The 3D camera (20) enters the standby state when the operation of the 3D camera (20) is turned off by the camera control unit (40) after the turning-on of the operation of the 3D camera (20) thereby.

Description

Image sensing device, image sensing method, and program

[0001] The present disclosure relates to an image sensing device, an image sensing method, and a program.

[0002] For example, Patent Literature 1 discloses a device for personal authentication or determining whether a person is present or absent, which uses both a 2D camera and a 3D camera, making it possible to prevent spoofing using photographs and to detect only targets within a predetermined distance range.

[0003] International Publication No. 2020 / 075525

[0004] However, since a 3D camera is equipped with a light-emitting device (light source) for measuring distance, when a 3D camera is used as in the device disclosed in Patent Document 1, power consumption increases due to the light emitted by the light source.

[0005] Therefore, the present disclosure provides an image sensing device and the like that can achieve low power consumption while using both a 2D camera and a 3D camera.

[0006] The image sensing device of the present disclosure comprises a 2D camera that generates a two-dimensional image, a 3D camera that has a light source and generates a distance image based on reflected light from the light source, an image recognition unit that uses the two-dimensional image or the distance image to recognize a subject appearing in the two-dimensional image, and a camera control unit that controls the on / off operation of the 3D camera, wherein the camera control unit turns on the operation of the 3D camera based on the results of the recognition using the two-dimensional image when the 3D camera has completed initialization processing after startup and is in a standby state where the light source is not emitting light, and the 3D camera enters the standby state when its operation is turned on by the camera control unit and then turned off by the camera control unit.

[0007] The image sensing method according to the present disclosure is an image sensing method executed by an image sensing device, the image sensing device comprising a 2D camera that generates a two-dimensional image, and a 3D camera having a light source that generates a distance image based on reflected light emitted by the light source, the image sensing method including an image recognition step that uses the two-dimensional image or the distance image to recognize a subject appearing in the two-dimensional image, and a camera control step that controls the on / off operation of the 3D camera, wherein in the camera control step, when the 3D camera has completed initialization processing after startup and is in a standby state in which the light source is not emitting light, the operation of the 3D camera is turned on based on the result of the recognition using the two-dimensional image, and the 3D camera enters the standby state when its operation is turned on in the camera control step and then turned off in the camera control step.

[0008] A program according to the present disclosure is a program that causes a computer to execute the image sensing method described above.

[0009] According to an image sensing device according to an aspect of the present disclosure, it is possible to achieve low power consumption while using a 2D camera and a 3D camera in combination.

[0010] FIG. 1 is a block diagram showing an example of an image sensing device according to a first embodiment. FIG. 2 is a sequence diagram showing a first example of operation of the image sensing device according to the first embodiment. FIG. 3 is a sequence diagram showing a second example of operation of the image sensing device according to the first embodiment. FIG. 4 is a sequence diagram showing a third example of operation of the image sensing device according to the first embodiment. FIG. 5 is a block diagram showing an example of an image sensing device according to a second embodiment. FIG. 6 is a block diagram showing an example of an image sensing device according to a third embodiment. FIG. 7 is a flowchart showing an example of an image sensing method according to other embodiments.

[0011] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0012] The embodiments described below are all comprehensive or specific examples, and the numerical values, shapes, materials, components, arrangement and connection of the components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure.

[0013] First Embodiment An image sensing device according to a first embodiment will be described below with reference to FIGS. 1 to 5. FIG.

[0014] Fig. 1 is a block diagram showing an example of an image sensing device 100 according to embodiment 1. Fig. 1 also shows an MPU (Micro Processing Unit) 200 that processes information output from the image sensing device 100. The MPU 200 may be a component of the image sensing device 100.

[0015] The image sensing device 100 is a device for recognizing a subject appearing in an image, and includes a 2D camera 10, a 3D camera 20, an image recognition unit 30, a camera control unit 40, a 2D output control unit 51, and a 3D output control unit 52. Note that, although the image recognition unit 30 is shown as a component included in the camera control unit 40 in Fig. 1, the image recognition unit 30 does not have to be a component of the camera control unit 40. Also, in Fig. 1, the 2D output control unit 51 and the 3D output control unit 52 are collectively referred to as the output control unit 50.

[0016] The image sensing device 100 is a computer including a processor, a memory, etc. The memory may be a read-only memory (ROM) or a random access memory (RAM), and can store programs executed by the processor. The image recognition unit 30, the camera control unit 40, and the output control unit 50 are realized by the processor, etc., that executes programs stored in the memory.

[0017] The 2D camera 10 is a camera that generates a two-dimensional image. The 2D camera 10 is, for example, a sensor such as an image sensor or an RGB sensor. Note that the two-dimensional image generated by the 2D camera 10 may be black and white.

[0018] For example, the 2D camera 10 includes a control unit 11. The control unit 11 is a processing unit that controls the focus and exposure of the 2D camera 10. Note that the control unit 11 may be provided in the image sensing device 100 and may not be provided in the 2D camera 10. For example, the control unit 11 controls the focus and exposure of the 2D camera 10 based on a control signal from the camera control unit 40.

[0019] The 3D camera 20 has a light source 21 and generates a distance image based on reflected light emitted by the light source 21. When the 3D camera 20 receives a signal to turn on its operation, it causes the light source 21 to emit light (e.g., intermittent light emission) at a predetermined timing and generates a distance image from the exposed signal. When the 3D camera 20 receives a signal to turn off its operation, it turns off the light emission of the light source 21 and stops generating the distance image. In this case, the light emission of the light source 21 is always off. Note that the 3D camera 20 may emit light from the light source 21 while receiving a signal to turn on its operation, and turn off the light emission of the light source 21 when the signal to turn on its operation is no longer received. The 3D camera 20 may be, for example, a camera using a time-of-flight (TOF) system or a light detection and ranging (LiDAR) system.

[0020] The 3D camera 20 consumes a lot of power because it needs the light source 21 to emit light in order to generate a distance image, whereas the 2D camera 10 does not basically need a light source and therefore consumes less power.

[0021] The image recognition unit 30 recognizes a subject in a two-dimensional image using a two-dimensional image generated by the 2D camera 10 or a distance image generated by the 3D camera 20. For example, recognition involves identifying whether a predetermined object is present in the two-dimensional image. The predetermined object is the target of recognition and is appropriately set depending on the application. For example, when a human face is recognized using the image sensing device 100, the predetermined object is the human face. For example, if a dog's face is present in the two-dimensional image, the recognition result indicates that the two-dimensional image does not contain a human face (although depending on the performance of the image recognition unit 30, it may indicate that a dog's face is present), and if a human face is present in the two-dimensional image, the recognition result indicates that the human face is present in the two-dimensional image. Note that the predetermined object is not limited to a human face.

[0022] The camera control unit 40 controls the on / off operation of the 3D camera 20. Specifically, when turning on the operation of the 3D camera 20, the camera control unit 40 outputs a signal to the 3D camera 20 to turn on the operation of the 3D camera 20, and when turning off the operation of the 3D camera 20, the camera control unit 40 outputs a signal to the 3D camera 20 to turn off the operation of the 3D camera 20. Note that when turning on the operation of the 3D camera 20, the camera control unit 40 may continue to output a signal to the 3D camera 20 to turn on the operation of the 3D camera 20, and when turning off the operation of the 3D camera 20, the camera control unit 40 may stop outputting the signal to turn on the operation of the 3D camera 20. When the 3D camera 20 is in a standby state, the camera control unit 40 turns on the operation of the 3D camera 20 based on the result of recognition using a two-dimensional image. The standby state of the 3D camera 20 is a state in which initialization processing after startup is completed and the light source 21 is always off. In other words, the standby state of the 3D camera 20 is a state in which the power of the 3D camera 20 is on but the light source 21 is not emitting light at all times. The 3D camera 20 enters the standby state when its operation is turned on by the camera control unit 40 and then turned off by the camera control unit 40. In this way, controlling the on and off of the operation of the 3D camera 20 is not intended to control the on and off of the power of the 3D camera 20.

[0023] For example, when the 3D camera 20 is in a standby state and the result of recognition using a two-dimensional image indicates that a predetermined object is captured in the two-dimensional image, the camera control unit 40 turns on the operation of the 3D camera 20. Furthermore, when the operation of the 3D camera 20 is on, the camera control unit 40 turns off the operation of the 3D camera 20 based on the result of recognition using a two-dimensional image or a distance image. Specific examples of the operation of the camera control unit 40 will be described later.

[0024] The 2D output control unit 51 controls the output of the 2D image based on the result of recognition using the 2D image or the distance image. For example, the 2D output control unit 51 extracts a portion of the 2D image based on the result of recognition and outputs it to the MPU 200.

[0025] The 3D output control unit 52 controls the output of the distance image based on the result of the recognition using the two-dimensional image or the distance image. For example, the 3D output control unit 52 extracts a portion of the distance image based on the result of the recognition and outputs it to the MPU 200.

[0026] The distance images from the 3D camera 20 can be very large data, which can lead to problems such as a decrease in the processing speed of the MPU 200, which analyzes the two-dimensional images and distance images or performs authentication using them, and an increase in power consumption due to an increase in the processing load. In response to this, the processing load on the MPU 200 can be reduced by controlling the output of only a portion of the two-dimensional images and distance images depending on the recognition results, for example. This can improve the processing speed of the MPU 200 and also suppress an increase in power consumption due to an increase in the processing load.

[0027] The MPU 200 is an output destination of the two-dimensional image and the distance image, and is a processing unit that analyzes the two-dimensional image and the distance image or performs authentication using them. For example, when the MPU 200 recognizes a subject in the two-dimensional image as a human face, it identifies individual differences in the face and performs facial authentication. Note that facial authentication is just one example, and the target of authentication by the MPU 200 is not particularly limited.

[0028] Next, a specific example of the operation of the image sensing device 100 will be described with reference to FIGS.

[0029] First, a first example of the operation of the image sensing device 100 will be described with reference to FIG.

[0030] FIG. 2 is a sequence diagram showing a first example of the operation of the image sensing device 100 according to the first embodiment.

[0031] First, the 2D camera 10 generates a two-dimensional image of the surroundings (step S101).

[0032] At this time, the 3D camera 20 is in a standby state, and the light source 21 is not emitting light (step S102).

[0033] The 2D camera 10 outputs the generated two-dimensional image (step S103). For example, the 2D camera 10 outputs the two-dimensional image to the image recognition unit 30 and the 2D output control unit 51. The 2D output control unit 51 also outputs the two-dimensional image to the MPU 200. In a first example, the 2D output control unit 51 outputs the two-dimensional image directly to the MPU 200. For example, the MPU 200 outputs the two-dimensional image to a display or the like.

[0034] The image recognition unit 30 uses the two-dimensional image to recognize the subject appearing in the two-dimensional image. Specifically, the image recognition unit 30 performs face recognition to identify whether a predetermined object (e.g., a human face) appears in the two-dimensional image (step S104). For example, when a person moves into the viewing angle of the 2D camera 10, the result of recognition using the two-dimensional image indicates that a human face appears in the two-dimensional image. For example, the predetermined object to be recognized is set by the MPU 200.

[0035] When the image recognition unit 30 detects a face in the two-dimensional image, it notifies the camera control unit 40 that a face has been detected (step S105).

[0036] Upon receiving notification that a face has been detected, the camera control unit 40 turns on the operation of the 3D camera 20 (step S106). For example, the camera control unit 40 outputs a signal to the 3D camera 20 to turn on the operation of the 3D camera 20. In this manner, when the 3D camera 20 is in a standby state, the camera control unit 40 turns on the operation of the 3D camera 20 if the result of recognition using a two-dimensional image indicates that a human face is captured in the two-dimensional image. In other words, the camera control unit 40 does not turn on the operation of the 3D camera 20 (in other words, does not cause the light source 21 to emit light) until the target of recognition appears in the two-dimensional image, and then turns on the operation of the 3D camera 20 when the target of recognition appears in the two-dimensional image, allowing the 3D camera 20 to generate a distance image. This allows the 3D camera 20 to be in a standby state (i.e., the light emission of the light source 21 is always off) until the target of recognition appears in the two-dimensional image, thereby achieving low power consumption.

[0037] Upon receiving a signal to turn on the operation of the 3D camera 20, the 3D camera 20 starts emitting light and exposing, generates a distance image of the surroundings (step S107), and outputs the image (step S108). For example, the 3D camera 20 outputs the distance image to the image recognition unit 30 and the 3D output control unit 52. The 3D output control unit 52 also outputs the distance image to the MPU 200. In a first example, the 3D output control unit 52 outputs the distance image directly to the MPU 200. For example, the MPU 200 performs facial recognition or the like using the two-dimensional image and the distance image.

[0038] The image recognition unit 30 uses the two-dimensional image or the distance image to recognize the subject appearing in the two-dimensional image. Specifically, the image recognition unit 30 performs face recognition to identify whether a predetermined object (e.g., a human face) is no longer appearing in the two-dimensional image (step S109). For example, if a person moves out of the viewing angle of the 2D camera 10, the recognition result using the two-dimensional image will indicate that the human face is no longer appearing in the two-dimensional image.

[0039] When the image recognition unit 30 does not detect a face in the two-dimensional image, it notifies the camera control unit 40 that a face has not been detected (step S110).

[0040] Upon receiving the notification that a face has no longer been detected, the camera control unit 40 turns off the operation of the 3D camera 20 (step S111). For example, the camera control unit 40 outputs a signal to the 3D camera 20 to turn off the operation of the 3D camera 20. In this way, while the 3D camera 20 is operating, the camera control unit 40 turns off the operation of the 3D camera 20 based on the result of recognition using a two-dimensional image or a distance image.

[0041] Then, the 3D camera 20 goes into standby mode again (step S112). In this way, if the face of the person to be recognized is no longer captured in the two-dimensional image after the operation of the 3D camera 20 is turned on, the target of recognition no longer exists, so the operation of the 3D camera 20 can be turned off (i.e., the 3D camera 20 can be put into standby mode). Accordingly, in the standby mode, the light emission of the light source 21 is always turned off, and the power consumption of the 3D camera 20 can be reduced.

[0042] Next, a second example of the operation of the image sensing device 100 will be described with reference to FIG.

[0043] FIG. 3 is a sequence diagram showing a second example of the operation of the image sensing device 100 according to the first embodiment.

[0044] The processing from step S201 to step S207 shown in FIG. 3 is the same as the processing from step S101 to step S107 shown in FIG. 2, and therefore a description thereof will be omitted.

[0045] The 3D camera 20 outputs the generated distance image (step S208). For example, the 3D camera 20 outputs the distance image to the image recognition unit 30 and the 3D output control unit 52.

[0046] After acquiring the distance image, the 3D output control unit 52 waits without outputting the distance image until it acquires a signal indicating whether or not the distance image can be output (step S209).

[0047] The image recognition unit 30 uses the two-dimensional image and the distance image to recognize the subject in the two-dimensional image. Specifically, the image recognition unit 30 performs face recognition to identify whether a predetermined object (e.g., a human face) is captured in the two-dimensional image (step S210). For example, when an image (e.g., a photograph) capturing a human face is within the viewing angle of the 2D camera 10, the result of the recognition using the two-dimensional image and the distance image indicates that the human face captured in the photograph is captured in the two-dimensional image. Because photographs are flat and real human faces have depth, the distance image can be used to identify whether the human face captured in the two-dimensional image is a real human face or a human face captured in a photograph.

[0048] When the image recognition unit 30 detects a photograph in the two-dimensional image, the image recognition unit 30 notifies the camera control unit 40 that a photograph has been detected (step S211).

[0049] Upon receiving the notification that a photograph has been detected, the camera control unit 40 outputs an output stop signal to the 3D output control unit 52 (step S212), which causes the 3D output control unit 52 to discard the distance image acquired in step S208 and to stop outputting the distance image to the MPU 200, thereby reducing the processing load on the MPU 200.

[0050] Furthermore, upon receiving notification that the photograph has been detected, the camera control unit 40 turns off the operation of the 3D camera 20 (step S213). For example, the camera control unit 40 outputs a signal to the 3D camera 20 to turn off the operation of the 3D camera 20. In this way, when the operation of the 3D camera 20 is on, the camera control unit 40 turns off the operation of the 3D camera 20 if the result of recognition using the two-dimensional image and the distance image indicates that the face of the person in the photograph is captured in the two-dimensional image.

[0051] Then, the 3D camera 20 enters the standby state again (step S214). In this way, after the operation of the 3D camera 20 is turned on, if it is determined from the distance image that the recognized face is the face of a person in the photograph, the recognized face is not the real person and is not the target of recognition, so the operation of the 3D camera 20 can be turned off (i.e., the 3D camera 20 can be placed in standby state). Accordingly, in the standby state, the light emission of the light source 21 is always turned off, and the power consumption of the 3D camera 20 can be reduced.

[0052] If no photograph is detected, the processes from step S109 onwards described in the first example are carried out.

[0053] Next, a third example of the operation of the image sensing device 100 will be described with reference to FIG.

[0054] FIG. 4 is a sequence diagram showing a third example of the operation of the image sensing device 100 according to the first embodiment.

[0055] The processing from step S301 to step S306 shown in Fig. 4 is the same as the processing from step S101 to step S106 shown in Fig. 2, and therefore description thereof will be omitted. Note that, in the first example, it has been explained that the 2D output control unit 51 outputs the two-dimensional image as is to the MPU 200, but in the third example, the 2D output control unit 51 controls the output of the two-dimensional image based on the result of recognition using the two-dimensional image or the distance image.

[0056] When a predetermined object (e.g., a human face) is captured in the two-dimensional image, the image recognition unit 30 outputs position information of the portion of the two-dimensional image in which the human face is captured to the 2D output control unit 51 and the 3D output control unit 52 (step S307). For example, the image recognition unit 30 outputs, as the position information, information indicating pixels corresponding to the area in the two-dimensional image that is detected as the human face to the 2D output control unit 51 and the 3D output control unit 52.

[0057] Upon receiving a signal to turn on the operation of the 3D camera 20, the 3D camera 20 starts emitting light and exposing, generates a distance image of the surroundings (step S308), and outputs it (step S309). For example, the 3D camera 20 outputs the distance image to the image recognition unit 30 and the 3D output control unit 52. Note that, in the first example, it was described that the 3D output control unit 52 outputs the distance image directly to the MPU 200, but in the third example, the 3D output control unit 52 controls the output of the distance image based on the result of recognition using a two-dimensional image or a distance image.

[0058] Based on the acquired position information, the 2D output control unit 51 outputs a 2D image of a portion of the 2D image in which a human face appears to the MPU 200 (step S310). For example, the 2D output control unit 51 extracts a pixel portion from the 2D image that corresponds to an area detected as a human face, and outputs the pixel portion in the 2D image to the MPU 200.

[0059] Based on the acquired position information, the 3D output control unit 52 also outputs a distance image of a portion of the two-dimensional image in which a human face appears to the MPU 200 (step S311). For example, the 3D output control unit 52 extracts a pixel portion from the distance image that corresponds to an area detected as a human face in the two-dimensional image, and outputs the pixel portion in the distance image to the MPU 200.

[0060] In this way, if the result of recognition using a two-dimensional image or a distance image indicates that a predetermined object (e.g., a human face) is captured in the two-dimensional image, the 2D output control unit 51 outputs a two-dimensional image of the portion of the two-dimensional image where the human face is captured, and the 3D output control unit 52 outputs a distance image of the portion of the two-dimensional image where the human face is captured. If the human face that is the target of recognition is captured in the two-dimensional image, it is possible to output only the two-dimensional image of the portion where the human face is captured (in other words, a two-dimensional image with the portion where the human face is not captured removed), and it is also possible to output only the distance image of the portion where the human face is captured (in other words, a distance image with the portion where the human face is not captured removed). This reduces the processing load on the MPU 200.

[0061] In addition, when multiple faces are detected in step S304 or when a human face in the two-dimensional image is far away, it may be desirable to authenticate a face within a certain distance. In such cases, the image recognition unit 30 outputs information about pixels corresponding to an area within a predetermined distance range among areas detected as a human face in the two-dimensional image to the 2D output control unit 51 and the 3D output control unit 52. As a result, in step S310, the 2D output control unit 51 can output a two-dimensional image of a portion of the two-dimensional image in which a human face within the predetermined distance range is displayed to the MPU 200. In addition, in step S311, the 3D output control unit 52 can output a distance image of a portion of the two-dimensional image in which a human face within the predetermined distance range is displayed to the MPU 200.

[0062] In this way, if the result of recognition using the two-dimensional image and the distance image indicates that a human face is captured within a predetermined distance range, the 2D output control unit 51 may output a two-dimensional image of the portion of the two-dimensional image where the human face is captured, and the 3D output control unit 52 may output a distance image of the portion of the two-dimensional image where the human face is captured. If the human face to be recognized is captured in the two-dimensional image and is within a predetermined distance range, only the two-dimensional image of this portion can be output, and only the distance image of this portion can be output. This reduces the processing load on the MPU 200.

[0063] Although not shown in FIG. 4, in the third example, the processing from step S109 onwards is performed in the same manner as in the first example, or the processing from step S209 onwards is performed in the same manner as in the second example.

[0064] Next, a fourth example of the operation of the image sensing device 100 will be described with reference to FIG.

[0065] FIG. 5 is a sequence diagram showing a fourth example of the operation of the image sensing device 100 according to the first embodiment.

[0066] The processing from step S401 to step S408 shown in FIG. 5 is the same as the processing from step S101 to step S108 shown in FIG. 2, and therefore a description thereof will be omitted.

[0067] The image recognition unit 30 recognizes the distance to the face of the person appearing in the two-dimensional image based on the distance image, and outputs this distance information to the camera control unit 40 (step S409), and the camera control unit 40 outputs this distance information to the 2D camera 10 (specifically, the control unit 11) (step S410).

[0068] The 2D camera 10 (control unit 11) controls the focus of the 2D camera 10 based on the acquired distance information (step S411), and outputs the two-dimensional image acquired by the focus-controlled 2D camera 10 (step S412).

[0069] In this way, the control unit 11 controls the focus of the 2D camera 10 based on the distance information to the face of the person shown in the two-dimensional image indicated by the distance image. The focus of the 2D camera 10 can be controlled at high speed based on the distance from the image sensing device 100 to the face of the person shown in the two-dimensional image, making it easier to focus on the face of the person shown in the two-dimensional image, thereby improving the accuracy of recognition or authentication using this two-dimensional image.

[0070] The control unit 11 may further control the exposure of the 2D camera 10 based on position information of a person's face appearing in the two-dimensional image.

[0071] Although not shown in FIG. 5, in the fourth example, the processing from step S109 onwards is performed in the same manner as in the first example, or the processing from step S209 onwards is performed in the same manner as in the second example.

[0072] As described above, the 3D camera 20 can be in a standby state until the desired recognition result (e.g., detection of a human face) is obtained using the two-dimensional image generated by the 2D camera 10. When the 3D camera 20 is in a standby state, it is activated (i.e., powered on), but the light source 21 is not emitting light, and only standby power is consumed. This reduces the power consumption of the 3D camera 20. Therefore, until the desired recognition result is obtained using the two-dimensional image, the light source 21 of the 3D camera 20 is not emitting light, thereby reducing power consumption. Once the desired recognition result is obtained, the operation of the 3D camera 20 can be turned on to illuminate the light source 21 for detailed recognition. After that, when the operation of the 3D camera 20 is turned off, the 3D camera 20 enters the standby state again, thereby reducing the power consumption of the 3D camera 20. Therefore, low power consumption can be achieved while using the 2D camera 10 and the 3D camera 20 in combination. Furthermore, in the image sensing device 100, the operation of the 3D camera 20 is controlled to be turned on and off based on the results of the recognition, but the power to the 3D camera 20 is not controlled to be turned on and off based on the results of the recognition. This prevents the initialization process from being performed each time the power to the 3D camera 20 is turned on and off, thereby reducing delays.

[0073] Second Embodiment Next, an image sensing device according to a second embodiment will be described with reference to FIG.

[0074] Fig. 6 is a block diagram showing an example of an image sensing device 100a according to embodiment 2. Fig. 6 also shows an MPU 200 that processes information output from the image sensing device 100a. The MPU 200 may be a component of the image sensing device 100a.

[0075] The image sensing device 100a differs from the image sensing device 100 in the first embodiment in that it includes a 2D camera 10a instead of the 2D camera 10, a plurality of 3D cameras instead of the 3D camera 20, and a camera control unit 40a instead of the camera control unit 40. The other points are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0076] The image sensing device 100a includes multiple 3D cameras, allowing it to sense a wider area than a single 3D camera. For example, the image sensing device 100a can be used for obstacle detection, which requires wide-area sensing.

[0077] 6, the image sensing device 100a includes multiple 3D cameras, such as 3D cameras 20a and 20b. The 3D camera 20a includes a light source 21a and generates a distance image based on the reflected light of light emitted by the light source 21a. The 3D camera 20b includes a light source 21b and generates a distance image based on the reflected light of light emitted by the light source 21b. Note that the image sensing device 100a may include three or more 3D cameras.

[0078] The 2D camera 10a is a camera that generates a two-dimensional image. The 2D camera 10a is, for example, a wide-viewing-angle camera. Since the image sensing device 100a includes multiple 3D cameras, the 2D camera 10a is also a wide-viewing-angle camera.

[0079] In addition to the functions of the camera control unit 40 in embodiment 1, the camera control unit 40a has a function of, when the multiple 3D cameras are in a standby state, determining at least one 3D camera from the multiple 3D cameras based on the position of a subject captured in a two-dimensional image, and activating the operation of the determined at least one 3D camera based on the result of recognition using the two-dimensional image. For example, if the image recognition unit 30 identifies that a subject captured in a two-dimensional image is within the viewing angle of the 3D camera 20a, the camera control unit 40a determines the 3D camera 20a from the 3D cameras 20a and 20b. Then, the camera control unit 40a activates the operation of the determined 3D camera 20a, for example, when the subject captured in the two-dimensional image is a predetermined object (e.g., a human face).

[0080] In this way, by turning on the operation of only the 3D camera 20a, which is capable of generating a distance image of the position of the subject captured in the two-dimensional image, it is possible to achieve lower power consumption than when all of the 3D cameras 20a and 20b are turned on. Furthermore, since it is possible to prevent distance images from all of the 3D cameras 20a and 20b from being output to the MPU 200, it is possible to reduce the processing load on the MPU 200. This improves the processing speed of the MPU 200 and also suppresses an increase in power consumption due to an increase in the processing load.

[0081] The image sensing device 100a in the second embodiment may also have the functions of the image sensing device 100 described in the first to fourth examples of the first embodiment.

[0082] Third Embodiment Next, an image sensing device according to a third embodiment will be described with reference to FIG.

[0083] Fig. 7 is a block diagram showing an example of an image sensing device 100b according to embodiment 3. Fig. 7 also shows an MPU 200 that processes information output from the image sensing device 100b. The MPU 200 may be a component of the image sensing device 100b.

[0084] The image sensing device 100b differs from the image sensing device 100 in the first embodiment in that it includes a BW-TOF camera 110 instead of the 2D camera 10 and the 3D camera 20. The other points are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0085] The BW-TOF camera 110 is a camera in which a 2D camera and a 3D camera are integrated together, in other words, the BW-TOF camera 110 is a camera that has both the functions of a 2D camera and a 3D camera.

[0086] The BW-TOF camera 110 includes a BW-TOF sensor 111 and a light source 112. The BW-TOF sensor 111 is provided with pixels (IR pixels) for generating a distance image and pixels (BW pixels) for generating a two-dimensional image (black and white image), and is capable of generating a two-dimensional image when the light source 112 is not emitting light, and is capable of generating both a distance image and a two-dimensional image based on the reflected light of the light emitted by the light source 112.

[0087] Although the BW-TOF camera 110 is shown here as an example of a camera in which a 2D camera and a 3D camera are integrated, there is no particular limitation as long as the camera is an integrated 2D camera and a 3D camera, in other words, a camera that has both the functions of a 2D camera and a 3D camera. For example, the integrated 2D camera and a 3D camera may be an RGB-TOF camera. An RGB-TOF camera is a camera that can acquire color images instead of black and white images, as opposed to the BW-TOF camera 110.

[0088] In this way, even when the 2D camera and the 3D camera are integrated, it is possible to achieve low power consumption while using both the 2D camera and the 3D camera together. Furthermore, by integrating the 2D camera and the 3D camera, it is possible to reduce the size of the image sensing device 100b.

[0089] The image sensing device 100b in the third embodiment may also have the functions of the image sensing device 100 described in the first to fourth examples of the first embodiment.

[0090] While the image sensing device according to one or more aspects of the present disclosure has been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to each embodiment and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects of the present disclosure.

[0091] For example, in the above embodiment, an example has been described in which the camera control unit turns off the operation of the 3D camera based on the result of recognition using a two-dimensional image or a distance image when the operation of the 3D camera is on, but this is not limiting. For example, the camera control unit may turn off the operation of the 3D camera based on the result of input from an external device when the operation of the 3D camera is on.

[0092] For example, the external device may be an MPU or the like, which uses the results of recognition using two-dimensional images or distance images to authenticate a subject appearing in the two-dimensional image. The MPU outputs a signal indicating that the authentication process is complete to the image sensing device. In this case, when the input result from the MPU to the image sensing device indicates that the authentication process is complete, no further recognition of the subject appearing in the two-dimensional image is required. Therefore, the operation of the 3D camera can be turned off, and the light source is also always turned off, thereby reducing the power consumption of the 3D camera.

[0093] For example, in the above embodiment, an example was described in which the image sensing device is equipped with a 2D output control unit that controls the output of two-dimensional images and a 3D output control unit that controls the output of distance images, but the image sensing device does not have to be equipped with a 2D output control unit and a 3D output control unit.

[0094] For example, in the above embodiment, an example has been described in which the image sensing device is provided with a control unit that controls the focus of the 2D camera, but the image sensing device does not necessarily have to be provided with such a control unit.

[0095] For example, the present disclosure can be realized not only as an image sensing device, but also as an image sensing method including steps (processing) performed by components that make up the image sensing device.

[0096] FIG. 8 is a flowchart illustrating an example of an image sensing method according to another embodiment.

[0097] The image sensing method is a method executed by an image sensing device, which includes a 2D camera that generates a two-dimensional image and a 3D camera that has a light source and generates a distance image based on reflected light from the light source. As shown in FIG. 8 , the image sensing method includes an image recognition step (step S11) that uses the two-dimensional image or the distance image to recognize a subject that appears in the two-dimensional image, and a camera control step that controls the on / off operation of the 3D camera. In the camera control step, when the 3D camera has completed initialization processing after startup and is in a standby state where the light source is not emitting light, the operation of the 3D camera is turned on based on the result of recognition using the two-dimensional image (step S12). The 3D camera enters a standby state when its operation is turned on in the camera control step and then turned off in the camera control step.

[0098] For example, the present disclosure can be realized as a program for causing a computer (processor) to execute steps included in an image sensing method. Furthermore, the present disclosure can be realized as a non-transitory computer-readable recording medium, such as a CD-ROM, on which the program is recorded.

[0099] For example, when the present disclosure is realized as a program (software), each step is performed by running the program using hardware resources such as a computer's CPU, memory, input / output circuits, etc. In other words, each step is performed by the CPU acquiring data from memory or input / output circuits, etc., performing calculations, and outputting the calculation results to memory or input / output circuits, etc.

[0100] In the above-described embodiments, each component included in the image sensing device may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0101] Some or all of the functions of the image sensing device according to the above embodiments are typically realized as an LSI, which is an integrated circuit. These may be individually integrated into single chips, or some or all of them may be integrated into a single chip. Furthermore, the integrated circuit is not limited to an LSI, and may be realized using a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor, which allows the connections and settings of circuit cells within an LSI to be reconfigured.

[0102] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0103] (Technology 1) An image sensing device comprising: a 2D camera that generates two-dimensional images; a 3D camera that has a light source and generates a distance image based on reflected light from the light emitted by the light source; an image recognition unit that uses the two-dimensional image or the distance image to recognize a subject that appears in the two-dimensional image; and a camera control unit that controls the on / off operation of the 3D camera, wherein the camera control unit turns on the operation of the 3D camera based on the results of the recognition using the two-dimensional image when the 3D camera has completed initialization processing after startup and is in a standby state where the light source is not emitting light; and the 3D camera enters the standby state when its operation is turned on by the camera control unit and then turned off by the camera control unit.

[0104] According to this, the 3D camera can be placed in a standby state until the desired recognition result is obtained using the two-dimensional image generated by the 2D camera. A 3D camera in a standby state is activated (i.e., powered on), but the light source is not emitting light, consuming only standby power. This reduces the power consumption of the 3D camera. Therefore, until the desired recognition result is obtained using the two-dimensional image, the 3D camera's light source is not illuminated, thereby reducing power consumption. Once the desired recognition result is obtained, the 3D camera's operation can be turned on to illuminate the 3D camera's light source for detailed recognition. When the 3D camera's operation is subsequently turned off, the 3D camera returns to a standby state, thereby reducing the power consumption of the 3D camera. Therefore, low power consumption can be achieved while using both a 2D camera and a 3D camera. Furthermore, in the image sensing device, the on / off of the 3D camera's operation is controlled based on the recognition result, but the on / off of the power supply of the 3D camera is not controlled based on the recognition result. This prevents initialization processing from being performed each time the 3D camera's power supply is repeatedly turned on and off, thereby reducing delays.

[0105] (Technology 2) The image sensing device described in Technology 1, wherein the camera control unit turns on the operation of the 3D camera when the result of the recognition using the two-dimensional image indicates that a predetermined object is captured in the two-dimensional image while the 3D camera is in the standby state.

[0106] This allows the 3D camera to be in a standby state until a predetermined object to be recognized appears in the two-dimensional image, thereby achieving low power consumption.

[0107] (Technology 3) An image sensing device described in Technology 1 or 2, wherein the camera control unit turns off the operation of the 3D camera when the 3D camera is on based on the results of the recognition using the two-dimensional image or the distance image.

[0108] This allows the operation of the 3D camera to be turned off based on the results of some recognition using a two-dimensional image or a distance image, which in turn keeps the light source off at all times, thereby reducing the power consumption of the 3D camera.

[0109] (Technology 4) The image sensing device described in Technology 3, wherein the camera control unit turns off the operation of the 3D camera when the 3D camera is on and the result of the recognition using the two-dimensional image or the distance image indicates that a predetermined object is not captured in the two-dimensional image.

[0110] For example, if a predetermined object that is the target of recognition is no longer captured in the two-dimensional image after the 3D camera is turned on, the target of recognition no longer exists, so the 3D camera can be turned off, and the light source will always be turned off, thereby reducing the power consumption of the 3D camera.

[0111] (Technology 5) An image sensing device described in Technology 3 or 4, wherein the camera control unit turns off the operation of the 3D camera when the 3D camera is on and the result of the recognition using the two-dimensional image and the distance image indicates that the two-dimensional image contains a predetermined object that is captured in the image.

[0112] For example, after the 3D camera is turned on, if it is determined from the distance image that the recognized object is the object shown in the image, the recognized object is not real and is not the target of recognition, so the 3D camera can be turned off, and the light source will also always be turned off, thereby reducing the power consumption of the 3D camera.

[0113] (Technology 6) An image sensing device described in any one of Technologies 1 to 5, wherein the camera control unit turns off the operation of the 3D camera based on input results from an external device when the 3D camera is on.

[0114] For example, the external device is a device that uses the results of recognition using two-dimensional images or distance images to authenticate a subject appearing in a two-dimensional image, and outputs a signal indicating that the authentication process is complete to the image sensing device. In this case, when the input result from the external device to the image sensing device indicates that the authentication process is complete, there is no further need to recognize the subject appearing in the two-dimensional image, so the operation of the 3D camera can be turned off, and accordingly, the light source emission is also always turned off, thereby reducing the power consumption of the 3D camera.

[0115] (Technology 7) The image sensing device described in any of Technologies 1 to 6 further includes a 2D output control unit that controls the output of the two-dimensional image based on the result of the recognition using the two-dimensional image or the distance image, and a 3D output control unit that controls the output of the distance image based on the result of the recognition using the two-dimensional image or the distance image.

[0116] Because distance images captured by a 3D camera can be very large, there are problems such as a decrease in the processing speed of a device to which the 2D images and distance images are output, which analyzes the 2D images and distance images or performs authentication using them, and an increase in power consumption due to an increase in processing load. In response to this problem, the processing load on the output device can be reduced by controlling the output of only a portion of the 2D images and distance images depending on the recognition results. This improves the processing speed of the output device and also suppresses an increase in power consumption due to an increase in processing load.

[0117] (Technology 8) An image sensing device described in Technology 7, wherein, when the result of the recognition using the two-dimensional image or the distance image indicates that a predetermined object is captured in the two-dimensional image, the 2D output control unit outputs a two-dimensional image of the portion of the two-dimensional image in which the predetermined object is captured, and the 3D output control unit outputs a distance image of the portion of the two-dimensional image in which the predetermined object is captured.

[0118] When a predetermined object to be recognized is included in a two-dimensional image, it is possible to output only the two-dimensional image of the portion where the object appears (in other words, the two-dimensional image with the portion where the object does not appear removed), and it is also possible to output only the distance image of the portion where the object appears (in other words, the distance image with the portion where the object does not appear removed).This reduces the processing load on the output device.

[0119] (Technology 9) An image sensing device described in Technology 7, wherein, when the result of the recognition using the two-dimensional image and the distance image indicates that a predetermined object is captured within a predetermined distance range, the 2D output control unit outputs a two-dimensional image of the portion of the two-dimensional image in which the predetermined object is captured, and the 3D output control unit outputs a distance image of the portion of the two-dimensional image in which the predetermined object is captured.

[0120] When a predetermined object to be recognized is captured in a two-dimensional image and the object is within a predetermined distance range, only the part of the two-dimensional image containing the object can be output, and only the part of the distance image containing the object can be output, thereby reducing the processing load on the output device.

[0121] (Technology 10) An image sensing device according to any one of technologies 1 to 9, further comprising a control unit that controls the focus of the 2D camera based on distance information to a subject appearing in the two-dimensional image indicated by the distance image.

[0122] This allows the focus of the 2D camera to be controlled quickly depending on the distance from the image sensing device to the subject appearing in the two-dimensional image, making it easier to focus on the subject appearing in the two-dimensional image, thereby improving the accuracy of recognition or authentication using this two-dimensional image.

[0123] (Technology 11) The image sensing device is provided with a plurality of the 3D cameras, and the camera control unit, when the plurality of 3D cameras are in the standby state, determines at least one of the plurality of 3D cameras based on the position of the subject appearing in the two-dimensional image, and turns on the operation of the determined at least one 3D camera based on the result of the recognition using the two-dimensional image. An image sensing device described in any one of Technologies 1 to 10.

[0124] For example, the image sensing device may be a device capable of sensing a wide area using multiple 3D cameras. In this case, by turning on only the 3D camera capable of generating a distance image of the position of a subject captured in a two-dimensional image, it is possible to achieve lower power consumption than when all of the multiple 3D cameras are turned on. Furthermore, it is possible to prevent distance images from all of the multiple 3D cameras from being output to the destination device, thereby reducing the processing load on the destination device. This improves the processing speed of the destination device and suppresses increases in power consumption due to increased processing load.

[0125] (Technology 12) The image sensing device according to any one of Technologies 1 to 11, wherein the 2D camera and the 3D camera are integrally provided.

[0126] In this way, by integrating the 2D camera and the 3D camera, it is possible to reduce the size of the image sensing device.

[0127] (Technology 13) An image sensing method executed by an image sensing device, the image sensing device comprising a 2D camera that generates a two-dimensional image, and a 3D camera that has a light source and generates a distance image based on reflected light of light irradiated by the light source, the image sensing method including: an image recognition step that uses the two-dimensional image or the distance image to recognize a subject appearing in the two-dimensional image; and a camera control step that controls on and off of the operation of the 3D camera, wherein in the camera control step, when the 3D camera has completed initialization processing after startup and is in a standby state where the light source is not emitting light, the operation of the 3D camera is turned on based on the result of the recognition using the two-dimensional image, and the 3D camera enters the standby state when its operation is turned on in the camera control step and then turned off in the camera control step.

[0128] This makes it possible to provide an image sensing method that can achieve low power consumption while using both a 2D camera and a 3D camera.

[0129] (Technology 14) A program that causes a computer to execute the image sensing method described in Technology 13.

[0130] This makes it possible to provide a program that can achieve low power consumption while using both a 2D camera and a 3D camera.

[0131] The present disclosure can be applied to devices that perform image recognition using a 2D camera and a 3D camera in combination.

[0132] 10, 10a 2D camera 11 Control unit 20, 20a, 20b 3D camera 21, 21a, 21b, 112 Light source 30 Image recognition unit 40, 40a Camera control unit 50 Output control unit 51 2D output control unit 52 3D output control unit 100, 100a, 100b Image sensing device 110 BW-TOF camera 111 BW-TOF sensor 200 MPU

Claims

1. A 2D camera that generates a two-dimensional image, A 3D camera that has a light source and generates a distance image based on the reflected light of the light irradiated by the light source, An image recognition unit that performs recognition on a subject captured in the two-dimensional image using the two-dimensional image or the distance image, A camera control unit that controls on and off of the operation of the 3D camera, and includes: When the 3D camera is in a standby state where the initialization process after startup is completed and the light source is not emitting light, the camera control unit turns on the operation of the 3D camera based on the result of the recognition using the two-dimensional image, After the operation of the 3D camera is turned on by the camera control unit, when the operation is turned off by the camera control unit, the 3D camera enters the standby state, An image sensing device.

2. When the 3D camera is in the standby state, if the result of the recognition using the two-dimensional image indicates that a predetermined object is captured in the two-dimensional image, the camera control unit turns on the operation of the 3D camera. The image sensing device according to Claim 1.

3. When the operation of the 3D camera is on, the camera control unit turns off the operation of the 3D camera based on the result of the recognition using the two-dimensional image or the distance image. The image sensing device according to Claim 1.

4. When the operation of the 3D camera is on, if the result of the recognition using the two-dimensional image or the distance image indicates that a predetermined object is not captured in the two-dimensional image, the camera control unit turns off the operation of the 3D camera. The image sensing device according to Claim 3.

5. When the operation of the 3D camera is on, if the result of the recognition using the two-dimensional image and the distance image indicates that a predetermined object captured in the image is captured in the two-dimensional image, the camera control unit turns off the operation of the 3D camera. The image sensing device according to Claim 3.

6. When the operation of the 3D camera is on, the camera control unit turns off the operation of the 3D camera based on the input result from an external device. The image sensing device according to Claim 1.

7. The image sensing device further includes: A 2D output control unit that controls the output of the two-dimensional image based on the result of the recognition using the two-dimensional image or the distance image; A 3D output control unit that controls the output of the distance image based on the result of the recognition using the two-dimensional image or the distance image, and comprising; The image sensing device according to any one of claims 1 to 6.

8. When the result of the recognition using the two-dimensional image or the distance image indicates that a predetermined object is shown in the two-dimensional image, The 2D output control unit outputs a two-dimensional image of a portion in the two-dimensional image where the predetermined object appears, The 3D output control unit outputs a distance image of a portion in the two-dimensional image where the predetermined object appears. The image sensing device according to claim 7.

9. When the result of the recognition using the two-dimensional image and the distance image indicates that a predetermined object is shown within a predetermined distance range, The 2D output control unit outputs a two-dimensional image of a portion in the two-dimensional image where the predetermined object appears, The 3D output control unit outputs a distance image of a portion in the two-dimensional image where the predetermined object appears. The image sensing device according to claim 7.

10. The image sensing device further comprises a control unit that controls the focus of the 2D camera based on the distance information to the subject shown in the two-dimensional image indicated by the distance image. The image sensing device according to any one of claims 1 to 6.

11. The image sensing device includes a plurality of the 3D cameras, When the plurality of 3D cameras are in the standby state, the camera control unit determines at least one of the plurality of 3D cameras based on the position of the subject shown in the two-dimensional image, and turns on the operation of the determined at least one 3D camera based on the result of the recognition using the two-dimensional image. The image sensing device according to any one of claims 1 to 6.

12. The 2D camera and the 3D camera are provided integrally. The image sensing device according to any one of claims 1 to 6.

13. An image sensing method executed by an image sensing device, wherein The image sensing device includes a 2D camera that generates a two-dimensional image, A 3D camera having a light source and generating a distance image based on the reflected light of the light irradiated by the light source, and The image sensing method includes an image recognition step of performing recognition on a subject shown in the two-dimensional image using the two-dimensional image or the distance image, and a camera control step of controlling on and off of the operation of the 3D camera, and in the camera control step, when the 3D camera is in a standby state where the initialization process after startup is completed and the light source is not emitting light, the operation of the 3D camera is turned on based on the result of the recognition using the two-dimensional image, after the operation of the 3D camera is turned on in the camera control step, when the operation is turned off in the camera control step, the 3D camera enters the standby state, An image sensing method.

14. A program for causing a computer to execute the image sensing method according to claim 13.