Image processing method and image processing apparatus

Abstract

To provide an image processing method for not outputting anything other than a specific object without a sense of incongruity.SOLUTION: An image processing method including obtaining a first input image from a camera, generating a background image on the basis of the first input image, determining whether the first input image includes a specific target object, determining whether the specific target object satisfies a predetermined position condition when the first input image includes the specific object, and replacing the specific target object that satisfies the predetermined position condition with the background image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] One embodiment of the present invention relates to an image processing method and an image processing apparatus for processing an image input from a camera.

Background Art

[0002] Patent Document 1 discloses a privacy image generation system that recognizes a person and blurs an image other than the recognized person.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since the system of Patent Document 1 blurs the background, it gives a sense of discomfort to the user. In addition, the system of Patent Document 1 may output an image without blurring a person other than the participant (specific object).

[0005] In consideration of the above circumstances, one aspect of the present disclosure aims to provide an image processing method for preventing a specific object from being output without a sense of discomfort.

Means for Solving the Problems

[0006] The image processing method acquires a first input image from a camera, generates a background image based on the first input image, determines whether a specific object is included in the first input image, and when the specific object is included in the first input image, determines whether the specific object satisfies a predetermined position condition, and replaces the specific object that satisfies the predetermined position condition with the background image.

Effects of the Invention

[0007] According to one embodiment of the present invention, it is possible to output only specific objects without causing any sense of incongruity. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of the image processing device 1. [Figure 2] This is a functional block diagram of the image processing device 1. [Figure 3] This is a flowchart showing the operation of the image processing method. [Figure 4] This flowchart shows an example of the operation of the background image generation unit 102. [Figure 5A] This figure shows an example of an image taken by camera 11. [Figure 5B] This figure shows an example of the output of the object determination unit 103. [Figure 6] This figure shows an example of the image after replacement. [Figure 7] This figure shows an example of an image taken by camera 11. [Figure 8] Figure 8(A) shows an example of the GUI displayed on the display unit 20, and Figure 8(B) shows an example of an image captured by the camera 11. [Figure 9] Figure 9(A) shows an example of the GUI displayed on the display unit 20, and Figure 9(B) shows an example of an image captured by the camera 11. [Modes for carrying out the invention]

[0009] Figure 1 is a block diagram showing the configuration of the image processing device 1. The image processing device 1 includes a camera 11, a CPU 12, a DSP 13, flash memory 14, RAM 15, a user interface (I / F) 16, a speaker 17, a microphone 18, a communication unit 19, and a display 20.

[0010] The camera 11, speaker 17, and microphone 18 are positioned, for example, above, below, left, or right, facing the display unit 20. The camera 11 acquires an image of the user standing in front of the display unit 20. The microphone 18 acquires the voice of the user standing in front of the display unit 20. The speaker 17 outputs sound to the user standing in front of the display unit 20.

[0011] The CPU 12 functions as a control unit that comprehensively controls the operation of the image processing device 1 by reading the operating program from the flash memory 14 into the RAM 15. Note that the program does not need to be stored in the device's own flash memory 14. The CPU 12 may download the program from, for example, a server and read it into the RAM 15 each time.

[0012] The DSP13 performs various processing on the images acquired by the camera 11, according to the control of the CPU 12. The DSP13 also performs various processing on the audio acquired by the microphone 18. However, audio processing is not an essential component of this invention.

[0013] The communication unit 19 transmits the video signal related to the image processed by the DSP 13 to another device. The communication unit 19 also transmits the audio signal processed by the DSP 13 to another device. This other device is, for example, an information processing device such as a PC at a remote end connected via the internet. The communication unit 19 also receives video and audio signals from the other device. The communication unit 19 outputs the received video signal to the display unit 20. The communication unit 19 outputs the received audio signal to the speaker 17. The display unit 20 displays the image acquired by the camera of the other device. The speaker 17 outputs the speaker's voice acquired by the microphone of the other device. As a result, the image processing device 1 functions as a communication system for voice conversation with a remote location.

[0014] Figure 2 is a block diagram showing the configuration of the image processing function, which is composed of a CPU 12 and a DSP 13. The image processing function includes an image acquisition unit 101, a background image generation unit 102, an object determination unit 103, a condition determination unit 104, and a replacement unit 105.

[0015] FIG. 3 is a flowchart showing the operation of the image processing method. The image acquisition unit 101 acquires an image (first input image) captured by the camera 11 (S11). The background image generation unit 102 generates a background image based on the first input image (S12). The object determination unit 103 determines whether or not a specific object (person) is included in the first input image (S13). The condition determination unit 104 determines whether or not the person satisfies a predetermined position condition when the person is included in the first input image (S14). The replacement unit 105 replaces the person who satisfies the predetermined position condition with the background image (S15).

[0016] FIG. 4 is a flowchart showing an example of the operation of the background image generation unit 102. The background image generation unit 102 first initializes the background image (S21). Thereafter, the background image generation unit 102 determines whether or not there is a temporal change for each pixel of the first input image (S22). That is, the background image generation unit 102 compares the first input image acquired at a first timing with a second input image acquired at a second timing different from the first timing. The background image generation unit 102 generates pixels that do not change in the first input image and the second input image as the background image (S23). The generated background image is output to the replacement unit 105.

[0017] When the background image generation unit 102 determines that there are pixels that change between the first input image and the second input image, it further determines whether or not the images of all the pixels have changed (S24). When the background image generation unit 102 determines that the images of all the pixels have changed, it returns the background image to the initialization (S24: Yes → S21). When the background image generation unit 102 determines that the images of all the pixels have not changed, it returns to the determination in S22. Note that the process in S24 may be determined not by all the pixels but by whether or not the number of changed pixels exceeds a predetermined value (predetermined ratio), for example, 50% or the like, with respect to the total number of pixels.

[0018] As a result, the area that does not change over time becomes the background image. Since the person moves, even if the pixels corresponding to the person at a certain timing do not become part of the background image, the person at that pixel moves to different pixels, so that the background image is generated for all pixels as time passes.

[0019] Note that the first input image and the second input image are not limited to the images of one frame each. Also, the first timing and the second timing do not necessarily need to be temporally continuous. The background image generation unit 102 may use, for example, an image obtained by averaging a plurality of frames as the first input image and the second input image. Of course, even if it is an averaged image, the first input image and the second input image correspond to images acquired at different timings respectively.

[0020] Next, the object determination unit 103 will be described. The object determination unit 103 determines whether a specific object (person) is included in the first input image. The object determination unit 103 specifies, for example, a plurality of pixels depicting one person by performing image segmentation processing. The image segmentation processing is a process of recognizing the boundary between a person and the background by using a predetermined algorithm using, for example, a neural network or the like.

[0021] FIG. 5A is a diagram showing an example of an image captured by the camera 11. In the example of FIG. 5A, the camera 11 is capturing the face images of a plurality of people standing along the longitudinal direction (depth direction) of the desk. The camera 11 is capturing four people on the left and right sides sandwiching the desk in the lateral direction, and a person at a position farther away from the desk.

[0022] The object detection unit 103 recognizes human pixels from the image captured by the camera 11. If the image captured by the camera 11 is the image shown in Figure 5A, the object detection unit 103 recognizes the boundaries between the bodies of the five people A1 to A5 and the background. As shown in Figure 5B, the object detection unit 103 generates labels (C1 to C5, C6) for each pixel that correspond to people A1 to A5 or the background, and outputs them to the replacement unit 105. Labels C1 to C5 are labels corresponding to people A1 to A5. Label C6 is a label corresponding to the background. The object detection unit 103 may also set bounding boxes, as shown by the rectangles B1 to B5 in the figure, at the positions of the faces of the recognized people. The object detection unit 103 may also output the position information of the bounding boxes to the replacement unit 105.

[0023] Furthermore, the object determination unit 103 determines the distance to each person based on the size of the bounding box. The flash memory 14 stores a table or function that shows the relationship between the size of the bounding box and the distance. The object determination unit 103 compares the set size of the bounding box with the table stored in the flash memory 14 to determine the distance to the person.

[0024] The object detection unit 103 may estimate the distance to a person from the total number of pixels with labels indicating a specific person, the maximum number of consecutive pixels vertically or horizontally with labels indicating a specific person. The object detection unit 103 may also estimate the person's body from the image captured by the camera 11 and estimate the person's position. The object detection unit 103 uses a predetermined algorithm, such as a neural network, to determine the person's skeleton (bones) from the image captured by the camera 11. Bones include the eyes, nose, neck, shoulders, and limbs. The flash memory 14 stores a table or function that shows the relationship between bone size and distance. The object detection unit 103 may compare the size of the recognized bones with the table stored in the flash memory 14 to determine the distance to the person.

[0025] Furthermore, the distance estimation method is not limited to the above example. For example, if camera 11 is a stereo camera (equipped with two or more cameras), the object determination unit 103 can determine the distance to each person based on the distance between the two cameras and the parallax between the two images. Alternatively, the object determination unit 103 may determine the distance to each person using a ranging mechanism such as LiDAR (Light Detection and Ranging).

[0026] The object determination unit 103 outputs identification information for each pixel (e.g., label information such as A1, A2) to the replacement unit 105, and outputs information indicating the 2D angle of each person and the distance to the camera to the condition determination unit 104. In other words, in this example, the object determination unit 103 outputs 3D position information as the position information of each person to the condition determination unit 104.

[0027] The condition determination unit 104 determines whether the location information of each person satisfies predetermined location conditions. The predetermined location conditions are, for example, when the distance value is greater than or equal to a predetermined value. In the example in Figure 5A, people A1, A2, A4, and A5 are close to the camera 11 and do not satisfy the predetermined location conditions. Person A3 is farther from the camera 11 and satisfies the predetermined location conditions.

[0028] Therefore, the condition determination unit 104 determines that person A3 satisfies the predetermined positional conditions and outputs information to the replacement unit 105 indicating that person A3 satisfies the positional conditions.

[0029] The replacement unit 105 replaces the image of a person that satisfies the positional conditions with the background image. In the example in Figure 5A, since person A3 satisfies the positional conditions, the pixel with label C3 corresponding to person A3 is replaced with a pixel of the background image generated by the background image generation unit 102. As a result, as shown in Figure 6, the pixel C3 with label corresponding to person A3 is replaced with a pixel of the background image. If the background image generation is not complete for the pixel to be replaced, the replacement unit 105 may output the image from the camera 11 as is, or replace it with a specific color. The replacement unit 105 may also average the color information of the pixels surrounding the pixel to be replaced and replace it with the average color. As a result, the replacement unit 105 can replace the image with one that does not look unnatural even if the background image has not been generated.

[0030] As described above, the image processing device 1 of this embodiment replaces distant figures with background images, thus preventing the output of images of people other than meeting participants without causing any sense of incongruity.

[0031] In the example described above, the background image generation unit 102 generated a background image using pixels that do not change over time. However, the background image generation unit 102 may also generate a background image using the determination result of the object determination unit 103. The background image generation unit 102 identifies areas in both the first input image and the second input image that do not contain people, and generates an image of the identified area as a background image. In this case, the background image generation unit 102 may use pixels from the first input image or pixels from the second input image.

[0032] Alternatively, if the first input image contains a region where a person is present, but the second input image does not, the background image generation unit 102 may generate a background image using pixels from that region in the second input image. Or, conversely, if the first input image does not contain a person, but the second input image does, the background image generation unit 102 may generate a background image using pixels from that region in the first input image.

[0033] In the above embodiment, the "predetermined position condition" was shown as a three-dimensional position condition including distance. However, the "predetermined position condition" is not limited to a three-dimensional position condition including distance. The condition determination unit 104 may, for example, use the fact that the field of view is within a predetermined angle as the predetermined position condition. In this case, the condition determination unit 104 may accept the setting of the field of view from the user. For example, the user may specify a range from 30 degrees to the left (-30 degrees) to 30 degrees to the right (+30 degrees) from the camera 11, with the front of the camera 11 being 0 degrees. In this case, the condition determination unit 104 will determine from the image of the camera 11 that a person in a pixel corresponding to the field of view from -30 degrees to +30 degrees does not satisfy the predetermined position condition. The condition determination unit 104 will also determine from the image of the camera 11 that a person in a pixel corresponding to outside the field of view range from -30 degrees to +30 degrees satisfies the predetermined position condition.

[0034] Alternatively, the condition determination unit 104 may accept the specification of a pixel range from the image of the camera 11. For example, as shown in Figure 7, the user specifies a certain pixel range S1 from the image of the camera 11. The condition determination unit 104 determines from the image of the camera 11 that people within the pixel range S1 do not meet the predetermined positional conditions. The condition determination unit 104 also determines from the image of the camera 11 that people in pixels outside the range of pixel range S1 meet the predetermined positional conditions. In the example in Figure 7, the condition determination unit 104 outputs information to the replacement unit 105 indicating that people A2, A3, and A4 meet the positional conditions. As a result, the replacement unit 105 replaces the pixels corresponding to people A2, A3, and A4 with the background image.

[0035] Thus, the image processing device 1 of this embodiment may accept the user's specification of positional conditions. Furthermore, the image processing device 1 of this embodiment may display a predetermined space on the display 20 and accept the user's specification of positional conditions for that predetermined space.

[0036] Figure 8(A) shows an example of a GUI displayed on the display unit 20. The CPU 12 of the image processing device 1 displays a predetermined space on the display unit 20 as shown in Figure 8(A). In this case, the CPU 12 functions as a display processing unit. In the example in Figure 8(A), the CPU 12 displays a two-dimensional planar image simulating a room on the display unit 20. The CPU 12 also displays planar images simulating a desk and a chair.

[0037] The CPU 12 accepts the specification of position conditions for a two-dimensional planar image via the user interface 16. The user interface 16 consists of a mouse, keyboard, or a touch panel superimposed on the display 20, and is an example of a reception unit. When the user selects an arbitrary position in the two-dimensional planar image shown in Figure 8(A), the CPU 12 accepts the pixel coordinates (two-dimensional coordinates) and distance information corresponding to the selected position as "position conditions". The CPU 12 also superimposes and displays an image indicating the selected position (a hatched image in the example of Figure 8(A)).

[0038] The left-right position of the two-dimensional planar image shown in Figure 8(A) corresponds to the X coordinate of the pixel coordinates. The up-down position of the two-dimensional planar image shown in Figure 8(A) corresponds to distance information. The Y coordinate of the pixel coordinates is accepted assuming that all pixels are selected. Alternatively, the Y coordinate of the pixel coordinates may be a numerical value (e.g., 0.7 to 2m) from the user that satisfies a predetermined position condition. Alternatively, as shown in Figure 8(B), the CPU 12 may superimpose and display an image (box S2) corresponding to the position condition in a predetermined space displayed on the display unit 20. In this case, the CPU 12 may accept an operation to change the vertical size of box S2. The CPU 12 associates the vertical size of box S2 with height information (Y coordinate range of the pixel coordinates).

[0039] The condition determination unit 104 determines from the image of the camera 11 whether each person satisfies predetermined positional conditions, according to the positional conditions received in the two-dimensional planar image of Figure 8(A). In the example of Figure 8(A), the condition determination unit 104 determines that person A3 satisfies the predetermined positional conditions and outputs information indicating that person A3 satisfies the positional conditions to the replacement unit 105. As a result, the replacement unit 105 replaces the pixels corresponding to person A3 with the background image.

[0040] Figure 9(A) shows another example of the GUI displayed on the display unit 20. The CPU 12 of the image processing device 1 displays a predetermined space on the display unit 20, similar to Figure 8(A).

[0041] The CPU 12 accepts the specification of position conditions for a two-dimensional plane image via the user interface 16. In the example shown in Figure 9(A), the CPU 12 superimposes an image of a sector corresponding to the position conditions onto the two-dimensional plane image and displays it on the display 20. The user can change the size of the sector. For example, the user can change the radius of the sector by touching and swiping the arc of the sector. The user can change the interior angle of the sector by touching and swiping the straight section of the sector.

[0042] The CPU 12 accepts field of view and distance information corresponding to the size of the sector as "position conditions". The radius of the sector image shown in Figure 9(A) corresponds to the distance information. The left and right opening angles of the sector image shown in Figure 9(A) correspond to the field of view and correspond to the X coordinate of the pixel coordinates. The Y coordinate of the pixel coordinates is accepted assuming that all pixels are selected. Alternatively, the Y coordinate of the pixel coordinates may be a numerical value of height information (for example, a value such as 0.7 to 2 mm) accepted by the user. Alternatively, as shown in Figure 9(B), the CPU 12 may superimpose and display an image corresponding to the sector (box S3) in a predetermined space displayed on the display unit 20. In this case, the CPU 12 may accept an operation to change the vertical size of box S3. The CPU 12 associates the vertical size of box S3 with the height information (Y coordinate of the pixel coordinates).

[0043] In this case as well, the condition determination unit 104 determines from the image of the camera 11 whether each person satisfies predetermined positional conditions, according to the positional conditions received in the two-dimensional planar image of Figure 9(A). In the example of Figure 9(A), the condition determination unit 104 determines that people A1, A3, and A5 satisfy the predetermined positional conditions and outputs information to the replacement unit 105 indicating that people A1, A3, and A5 satisfy the positional conditions. As a result, the replacement unit 105 replaces the pixels corresponding to people A1, A3, and A5 with the background image.

[0044] The condition determination unit 104 may determine whether the entire range of the bounding box corresponding to each person's image satisfies the predetermined position conditions, or it may determine whether a part of the bounding box satisfies the predetermined position conditions. When the condition determination unit 104 determines whether a part of the bounding box satisfies the predetermined position conditions, it may determine that the person in that bounding box satisfies the predetermined position conditions if the number of pixels that satisfy the position conditions exceeds a predetermined value (a predetermined percentage) relative to the total number of pixels in the bounding box, for example, 50%.

[0045] The description of this embodiment should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, rather than by the embodiments described above. Furthermore, the scope of the present invention includes the scope equivalent to the claims.

[0046] For example, the specific image is not limited to people. The specific image could be, for example, an animal, a PC screen, or a paper document. For example, even if a PC screen or paper document contains confidential information unrelated to the meeting, the image processing device of this embodiment replaces the specific image containing this confidential information with a background image. As a result, the image processing device of this embodiment can output an image that does not feel out of place while maintaining privacy. [Explanation of Symbols]

[0047] 1…Image processing device 11…Camera 12…CPU 13…DSP 14…Flash memory 15...RAM 16…User Interface 17...Speaker 18... Mike 19… Communications Department 20...Indicator 101...Image acquisition unit 102... Background image generation unit 103...Object determination unit 104…Condition judgment section 105...Replacement part

Claims

1. The camera acquires a first input image at a first timing and a second input image at a second timing different from the first timing. The first input image and the second input image are compared, and a background image is generated using the region that remains unchanged. Determine whether or not a specific object is included in the first input image. If the first input image includes the specific object, it is determined whether the specific object satisfies predetermined positional conditions. The specific object that satisfies the predetermined positional conditions is replaced with the background image. If the generation of the background image is not complete, the first input image is output. Image processing methods.

2. The aforementioned predetermined positional condition is a three-dimensional positional condition including distance. The image processing method according to claim 1.

3. If the aforementioned distance exceeds a predetermined value, it is determined that the position condition is met. The image processing method according to claim 2.

4. The aforementioned specific image includes an image of a person. The image processing method according to any one of claims 1 to 3.

5. Based on the size of the aforementioned specific image, it is determined whether or not the aforementioned specific positional conditions are met. The image processing method according to any one of claims 1 to 4.

6. The system accepts the specification of the aforementioned location conditions from the user. The image processing method according to any one of claims 1 to 5.

7. The display shows a predetermined space, The system accepts the designation of the positional conditions for the predetermined space. The image processing method according to claim 6.

8. The image corresponding to the positional conditions is superimposed and displayed in the predetermined space. The image processing method according to claim 7.

9. An image acquisition unit that acquires a first input image from a camera at a first timing and a second input image at a second timing different from the first timing, A background image generation unit compares the first input image and the second input image and generates a background image using regions that have not changed, An object determination unit that determines whether or not a specific object is included in the first input image, If the first input image includes the specific object, a condition determination unit determines whether the specific object satisfies predetermined positional conditions, A replacement unit that replaces the specific object that satisfies the predetermined positional conditions with the background image, Equipped with, The replacement unit outputs the first input image if the generation of the background image is not yet complete. An image processing device.

10. The aforementioned predetermined positional condition is a three-dimensional positional condition including distance. The image processing apparatus according to claim 9.

11. The condition determination unit determines that the position condition is met when the distance exceeds a predetermined value. The image processing apparatus according to claim 10.

12. The aforementioned specific image includes an image of a person. The image processing apparatus according to any one of claims 9 to 11.

13. The condition determination unit determines whether or not the specific positional condition is met based on the size of the specific image. The image processing apparatus according to any one of claims 9 to 12.

14. A reception unit is provided to receive the location conditions specified by the user. The image processing apparatus according to any one of claims 9 to 13.

15. The display unit is equipped with a display processing unit that displays a predetermined space. The reception unit receives the designation of the positional conditions for the predetermined space. The image processing apparatus according to claim 14.

16. The display processing unit superimposes and displays an image corresponding to the positional conditions in the predetermined space. The image processing apparatus according to claim 15.

Images