Image processing system and image processing method

Abstract

The purpose of the present invention is to reduce the time required for object extraction and cut-out processing.　The present invention is a video processing system comprising: a software processing unit that detects an object included in at least a partial input image included in input video and extracts the outline of the object; and a hardware processing unit that generates mask information for cutting out the object from each input image included in the input video using the outline extracted by the software processing unit, wherein the software processing unit and the hardware processing unit perform processing independently and in parallel.

Description

Technical Field

[0001] The present disclosure relates to a video processing technology for extracting an object of interest, such as a person, from a video captured by a camera or the like from a background.

Background Art

[0002] In communication tools using real-time video and audio used in web conferences and the like, a technology for extracting a video from a person and compositing it with another background is used. Such a cutting technology realizes communication not restricted by location by hiding a background that is not originally intended to be shown, and enables smoother communication by replacing it with a background suitable for communication. Various methods are known for such object extraction and cutting processing.

[0003] Classically, there are a region division method for dividing an image into a plurality of regions using feature amounts to extract an object, a region expansion method for searching for adjacent similar regions from a starting pixel and expanding the region, a division combination method for combining a region division method and a region expansion method, a contour method for extracting a contour line, an optical flow for extracting a moving region, etc. (see, for example, Non-Patent Document 1). As another approach, methods that mimic human thinking, such as fuzzy theory, deep learning, and genetic algorithms, are also well known (see, for example, Non-Patent Document 2).

[0004] For communication using real-time video and audio, video and audio processing such as object extraction and cutting processing of a person or the like is important. This makes it possible to enable smoother communication regardless of location and by compositing with an appropriate background or the like. The above video processing needs to be executed in processing time to satisfy the requirement of real-time performance of communication using the video and audio.

[0005] For example, if we consider remote ensemble performance as real-time video and audio communication, and assume a delay of up to 1 / 10th of a beat per beat for a song at 240 BPM (beats per minute), then the time for one beat is 60 seconds / 120 BPM = 0.25 seconds, and 1 / 10 of that is 0.025 seconds, or about 25 milliseconds. Therefore, to meet the real-time requirement, it is desirable to execute the process with a processing time of less than 25 milliseconds.

[0006] This 25 milliseconds time includes everything from the moment the subject moves to the shutter release, the processing time within the camera, the transmission time over the network, and the video and audio processing time within the communication system itself.

[0007] Of these, the object extraction and cropping processes are included in the video and audio processing time, which also includes processing for splitting and displaying the video. Therefore, the processing time available for object extraction and cropping is estimated to be a few milliseconds or less.

[0008] The aforementioned object extraction and cropping process includes receiving and processing image data for one frame of video. For example, if the video data is 60 frames per second, a data reception time of 1 / 60 second = 16.7 milliseconds is required, in addition to data processing time. Existing studies have reported processing times of several tens of milliseconds or more (see, for example, Non-Patent Document 3). Therefore, this does not meet the processing time requirements for the aforementioned object extraction and cropping process.

[0009] Therefore, in real-time video and audio communication in scenarios with strict latency requirements, such as remote ensemble performances, object extraction and cutout processing cannot be performed, hindering smooth communication that would otherwise require compositing with an appropriate background. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] Freixenet, Jordi, et al. “Yet another survey on image segmentation: Region and boundary information integration.” European conference on computer vision. Springer, Berlin, Heidelberg, 2002. [Non-Patent Document 2] Chouhan, Siddharth Singh, Ajay Kaul, and Uday Pratap Singh. “Soft computing approaches for image segmentation: a survey.” Multimedia Tools and Applications 77.21 (2018): 28483-28537 [Non-Patent Document 3] Ryu, Sangwoo, Kyungchan Ko, and James Won-Ki Hong. “Performance Analysis of Applying Deep Learning for Virtual Background of WebRTC-based Video Conferencing System.” 2021 22nd Asia-Pacific Network Operations and Management Symposium (APNOMS). IEEE, 2021. [Overview of the project] [Problems that the invention aims to solve]

[0011] This disclosure aims to reduce the time required for object extraction and cropping processes. [Means for solving the problem]

[0012] In this disclosure, a software processing unit performs advanced object detection and contour extraction, while a hardware processing unit generates mask information for cropping. Furthermore, by performing these processes in a pipeline, it is possible to reduce the processing time for object extraction and cropping.

[0013] The video processing system disclosed herein is A software processing unit that detects objects contained in at least some of the input images and extracts the contours of those objects, A hardware processing unit generates mask information for cutting out the object from each input image included in the input video using contours extracted by the software processing unit, Equipped with, The software processing unit and the hardware processing unit perform processing independently and in parallel.

[0014] The video processing method disclosed herein is The software processing unit detects objects contained in at least a portion of the input video and extracts the contours of those objects. The hardware processing unit generates mask information for cutting out the object from each input image included in the input video, using the contour extracted by the software processing unit. Equipped with, The software processing unit and the hardware processing unit perform processing independently and in parallel.

[0015] The software processing unit may extract the contour of the object using the first input image included in the input video, and the hardware processing unit may generate mask information for the second input image that arrives after the first input image included in the input video by correcting the contour extracted from the first input image or the mask information generated from the first input image. In this case, the hardware processing unit may perform the correction for each predetermined line section of each input image included in the input video.

[0016] The mask information may include contour information that can identify the contour of the object in any input image included in the input video. The contour information may include coordinates included in the contour of the object in any input image included in the input video, or may include vectors indicating the contour of the object in any input image included in the input video. Further, the mask information may be a mask image that covers areas other than the object in any input image included in the input video.

[0017] The hardware processing unit may generate a composite image in which areas other than the object are different in each input image included in the input video as the mask information.

[0018] Note that the above disclosures can be combined as much as possible.

Effect of the Invention

[0019] The present disclosure can reduce the time required for object extraction and cutting processes. Therefore, the present disclosure can perform object extraction and cutting processes and enable smooth communication by synthesizing with an appropriate background or the like in real-time video and audio communication in scenes with strict delay requirements such as remote ensemble.

Brief Description of the Drawings

[0020] [Figure 1] Shows a configuration example of the video processing system of the present disclosure. [Figure 2] It is a diagram for explaining the processing in the software processing unit. [Figure 3] It is a diagram for explaining the processing in the hardware processing unit. ​​​​​​​​​ [Figure 7] This diagram illustrates each step in the process of generating a mask image. [Modes for carrying out the invention]

[0021] Embodiments of this disclosure will be described in detail below with reference to the drawings. However, this disclosure is not limited to the embodiments shown below. These examples are illustrative, and this disclosure can be implemented in various modified and improved forms based on the knowledge of those skilled in the art. In this specification and in the drawings, components with the same reference numerals refer to the same components.

[0022] (First embodiment) Figure 1 shows an example of the configuration of the video processing system of this disclosure. The video processing system 10 of this disclosure extracts objects contained in each image (sometimes referred to as the input image) of each screen (frame) included in the input video, replaces the images of each screen (frame) with the extracted images (sometimes referred to as the composite image), and outputs them as output video. The video processing system 10 of this disclosure performs these object extraction and extraction processes through cooperative processing between a software processing unit 11 and a hardware processing unit 12. The hardware processing unit 12 can use an FPGA (Field Programmable Gate Array).

[0023] The video processing method disclosed herein is The software processing unit 11 detects objects contained in at least a portion of the input images and extracts the contours of those objects. The hardware processing unit 12 generates mask information for cutting out the object from each input image included in the input video using the contour extracted by the software processing unit 11, Equipped with, The software processing unit 11 and the hardware processing unit 12 perform processing independently in parallel.

[0024] Here, the mask information is any information that allows an object to be cut out from the input image, and may include contour information that can identify the contour of the object. For example, the mask information may include coordinates that indicate at least a part of the contour of the object, or it may include a vector that indicates the contour of the object. In this embodiment, as an example of mask information, an example of a mask image that covers the area of ​​the input image other than the object is shown.

[0025] The image processing system 10 may be a single device or may consist of multiple devices. For example, the image processing system 10 may have a software processing unit 11 and a hardware processing unit 12 that are physically separate. In this case, even if the software processing unit 11 and the hardware processing unit 12 are located in remote locations, the system of this disclosure can be configured by transmitting object contour information via an information transmission medium such as a communication network.

[0026] Furthermore, the software processing unit 11 can be implemented using a computer and a program, and the program can be recorded on a recording medium or provided via a network. The video processing program of this disclosure causes the computer to function as the software processing unit 11, and causes the software processing unit 11 and the hardware processing unit 12 to perform processing independently and in parallel.

[0027] As shown in Figure 2, the software processing unit 11 performs advanced object detection and contour extraction of object Ob(t) on image Io(t) at any point in time t included in the video. This provides contour information necessary for cutting out object Ob(t). The software processing unit 11 passes this contour information to the hardware processing unit 12. In this specification, image Io(t) at any point in time t included in the video may be referred to as the input image.

[0028] The algorithms used for detecting object Ob(t) and extracting the contours of object Ob(t) are not limited. The software processing unit 11 may process every image Io(t) of the video, or it may process every few images Io(t).

[0029] The hardware processing unit 12 uses contour information from the software processing unit 11 to generate a mask image Im(t) from the image Io(t) in which the region of object Ob(t) is transparent, as shown in Figure 3. Then, the hardware processing unit 12 overlays the mask image Im(t) on a layer above the image Io(t). This generates a composite image Ic(t) which is a combination of the image of object Ob(t) and the mask image Im(t).

[0030] Here, the areas in the mask image Im(t) other than the object Ob(t) may be plain, but they can be any image. For example, the hardware processing unit 12 may perform a compositing process with a background image different from the background of the image Io(t). The hardware processing unit 12 may also output mask information and / or an image of the object Ob(t).

[0031] This disclosure has the following advantages by having a software processing unit 11. • It can perform advanced object detection and contour extraction processes for objects (Ob(t)) that are difficult to implement with hardware processing. • The algorithms for object detection and contour extraction, which are difficult to implement with hardware processing, can be easily modified.

[0032] This disclosure has the following advantages by having a hardware processing unit 11. • It enables low-latency processing that is impossible to achieve with software processing.

[0033] This disclosure has the following advantages by having both a software processing unit 11 and a hardware processing unit 12. The aforementioned advantages of the software processing unit 11 and the hardware processing unit 12 can be taken advantage of as they are. Compared to implementing this in a separate processing unit, the implementation circuit size of the hardware processing unit 12 can be minimized, making implementation in devices easier.

[0034] (Second embodiment) In the video, as shown in Figure 4, the object Ob(t) in image Io(t) changes to Ob(t+δ) in image Io(t+δ). Therefore, in this embodiment, the hardware processing unit 12 uses arbitrary information generated by either the software processing unit 11 or the hardware processing unit 12, or both, when generating mask information. Specifically, the mask image Im(t+δ) at time t+δ is generated by correcting the contour information or mask image Im(t) at time t.

[0035] The hardware processing unit 12 corrects either the contour information at time t or the mask image Im(t) for every n horizontal lines (several to several hundred) of the input video image Io(t+δ) based on contour information from either the software processing unit 11 or the hardware processing unit 12, or both, generates a new mask image Im(t+δ), and outputs a composite image Ic(t+δ) output video in which only the object Ob(t+δ) is extracted from the image Io(t+δ).

[0036] The method of correcting contour information is not specified. Alternatively, correction of the mask image Im may be performed instead of correcting the contour information.

[0037] Referring to Figure 5, the processing flow for image data of one screen (frame) of the video will be explained. At time t1, the software processing unit 11 performs contour extraction processing of object Ob(t1) on image Io(t1) of frames k1-n and passes the contour information to the hardware processing unit 12. At time t2, the software processing unit 11 processes image Io(t2) of frames k2-n. Time t2 is, for example, after the software processing unit 11 has finished processing image Io(t1). However, this disclosure is not limited thereto. For example, the software processing unit 11 may periodically perform processing and update the contour information. Also, for example, the software processing unit 11 may perform processing in parallel and update the contour information.

[0038] The hardware processing unit 12 performs processing using the latest contour information from the software processing unit 11. For example, at time t1+δ1, the hardware processing unit 12 corrects either the contour information of the k1-n frame image Io(t1) or the mask image Im(t1) from the software processing unit 11, or both, for the k1 frame image Io(t1+δ1) input, and performs the process of generating the mask image Im(t1+δ1) and the composite image Ic(t1+δ1).

[0039] The arrival time t1+δ2 of the k1+1 frame is after the time t2 when the software processing unit 11 started processing the image Io(t2) of the k2-n frames. In this case, the software processing unit 11 has not yet finished processing the image Io(t2), and neither the contour information nor the mask image Im(t2) or both have been updated. Therefore, the hardware processing unit 12 can use either the contour information or the mask image Im(t1) extracted by the software processing unit 11 in the k1-n frames, which was used in the processing of the k1 frame, for the processing of the k1+1 frame.

[0040] Here, in the correction performed by the hardware processing unit 12, information processed in any past frame generated by the hardware processing unit 12 can be used. For example, in the hardware processing for frame k1+1, not only contour information extracted in frames k1-n may be used, but also mask information such as a mask image generated in the hardware processing of frame k1 may be used.

[0041] The same processing is performed for k2-n, k2, and k2+1 frames. This pipeline processing minimizes the delay between video input and video output for a given frame in the hardware processing unit 12.

[0042] (Third embodiment) Figure 6 shows an example of a method for generating the mask image Im(t+δ) in the second embodiment. In this embodiment, an example of a correction processing procedure using optical flow is shown with reference to Figure 7.

[0043] Step S101 The software processing unit 11 detects the object Ob(t) in the image Io(t) at time t and extracts the contour of the object Ob(t) (S101). This generates contour information for the object Ob(t) and passes the contour information to the hardware processing unit 12.

[0044] Step S102 The hardware processing unit 12 extracts minute cells around the boundary of object Ob(t) from image Io(t) based on contour information.

[0045] Step S103 The hardware processing unit 12 calculates the location and amount of movement of each minute cell extracted from object Ob(t) by detecting areas with high similarity from image Io(t+δ). Specifically, similarity can be detected by performing a correlation operation on pixels in the vicinity of the original position of the minute cell in image Io(t+δ).

[0046] Step S104 The hardware processing unit 12 can correct the mask image Im(t) at time t based on the location and amount of movement of object Ob(t), and generate a new mask image Im(t+δ).

[0047] Here, the extraction of minute cells can be performed sequentially for each minute line section without waiting for the arrival of image data for one screen (frame) in the video data. Here, the minute lines can be set to any n lines predetermined. The minute line sections may overlap; that is, there may be overlaps. Furthermore, although an example using optical flow is shown in this embodiment, other methods such as region expansion methods may also be used in this disclosure.

[0048] By performing processing for each minute line section, the waiting time for the arrival of image data for one screen (frame) can be reduced, thereby reducing processing delay.

[0049] (effect) This disclosure, by having both a software processing unit 11 and a hardware processing unit 12, can achieve both the advantages of the software processing unit 11, such as the ability to easily change advanced object detection and contour extraction processing and its algorithm, and the advantages of the hardware processing unit 12, such as low-latency processing that cannot be achieved with software processing alone. Furthermore, compared to implementing it with a single processing unit, the implementation circuit size of the hardware processing can be minimized, making it easier to implement in a device. • By using a pipeline processing method between the software processing unit 11 and the hardware processing unit 12, the delay from video input to video output for a given frame in the hardware processing unit 12 can be minimized. In real-time video and audio communication in scenarios with strict latency requirements, such as remote ensemble performances, object extraction and cutout processing can be performed and combined with appropriate backgrounds to achieve smooth communication.

[0050] As described above, this disclosure implements cooperative processing between a software processing unit 11 and a hardware processing unit 12. In particular, the software processing unit 11 performs advanced object detection and contour extraction processing, and the hardware processing unit 12 performs correction processing and other operations to generate mask information for cutout. Furthermore, by performing these processes in a pipeline, the processing time is reduced. This makes it possible to perform object extraction and cutout processing and synthesize them with an appropriate background in real-time video and audio communication in scenes with strict latency requirements, such as remote ensemble performances, thereby enabling smooth communication. [Explanation of symbols]

[0051] 10: Video Processing System 11: Software Processing Section 12: Hardware Processing Unit

Claims

1. A software processing unit that detects objects contained in at least some of the input images and extracts the contours of those objects, A hardware processing unit generates mask information for cutting out the object from each input image included in the input video using contours extracted by the software processing unit, Equipped with, The software processing unit extracts the contour of the object using the first input image included in the input video. The hardware processing unit generates mask information for a second input image that arrives after the first input image included in the input video by correcting the contour extracted from the first input image or the mask information generated from the first input image. The software processing unit and the hardware processing unit perform processing independently in parallel. Video processing system.

2. The hardware processing unit performs the correction for each predetermined line section of each input image included in the input video. The image processing system according to claim 1.

3. The mask information includes contour information that can identify the contour of the object in any input image included in the input video. The image processing system according to claim 1.

4. The mask information is a mask image that covers areas other than the object in any input image included in the input video. The image processing system according to claim 1.

5. The hardware processing unit generates a composite image in which, as mask information, the regions other than the object differ in each input image included in the input video. The image processing system according to claim 1.

6. The software processing unit detects objects contained in at least a portion of the input video and extracts the contours of those objects. The hardware processing unit generates mask information for cutting out the object from each input image included in the input video, using the contour extracted by the software processing unit. Equipped with, The software processing unit extracts the contour of the object using the first input image included in the input video. The hardware processing unit generates mask information for a second input image that arrives after the first input image included in the input video by correcting the contour extracted from the first input image or the mask information generated from the first input image. The software processing unit and the hardware processing unit perform processing independently in parallel. Image processing methods.

7. The software processing unit detects objects contained in at least a portion of the input video and extracts the contours of those objects. The hardware processing unit generates mask information for cutting out the object from each input image included in the input video, using the contour extracted by the software processing unit. Equipped with, The software processing unit extracts the contour of the object using the first input image included in the input video. The hardware processing unit generates mask information for a second input image that arrives after the first input image included in the input video by correcting the contour extracted from the first input image or the mask information generated from the first input image. The software processing unit and the hardware processing unit are made to perform processing independently and in parallel. Video processing program.

Images