Image processing system, program, and image processing method

Abstract

The present invention appropriately calculates a mask region.　Provided is an image processing system equipped with a mask region calculation unit, wherein the mask region calculation unit acquires a detection result of image detection processing for an image of a frame within a moving image, the mask region calculation unit calculates a mask region for an image of a first frame on the basis of the detection result of at least an image of a second frame, a detection frame or a detection region is associated with the at least an image of the second frame by the image detection processing, a detection score is set for the detection frame or the detection region, the detection result includes the detection frame or the detection region, and the detection score, and the second frame is a frame at a point in time different from the first frame on the time axis of the moving image.

Description

[Technical Field]

[0001] The present invention relates to an image processing system, a program, and an image processing method. [Background technology]

[0002] Patent Document 1 discloses a technique for applying masking to video. In Patent Document 1, a face is detected and a masking process is applied to the face image to protect personal privacy. However, face recognition is not always successful in all frames of a video. Therefore, Patent Document 1 employs several image processing controls (such as masking the predicted face position, stopping distribution, distributing an alternative image, or excluding frames for which detection has failed) when face detection fails. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. WO2016 / 088583 Summary of the Invention [Problem to be solved by the invention]

[0004] In image detection processing, a detection score is often output along with a detection frame, etc. The detection score indicates the accuracy (likelihood) of detecting the target. Patent Document 1 does not recognize the use of detection scores, and the usefulness of detection scores is overlooked. The inventors of the present application have discovered a new technology that can utilize detection scores to appropriately calculate mask areas even in situations where image detection is not necessarily successful.

[0005] The present invention provides an image processing system, a program, and an information processing method that are capable of appropriately calculating a mask area. [Means for solving the problem]

[0006] According to the present disclosure, the following inventions are provided. [1] An image processing system including a mask area calculation unit, wherein the mask area calculation unit obtains a detection result of an image detection process for an image of a frame in a moving image, the mask area calculation unit calculates a mask area for an image of a first frame based on the detection result of an image of at least a second frame, the image detection process associates a detection frame or detection area with the image of at least the second frame, and a detection score is set for the detection frame or the detection area, the detection result includes the detection frame or the detection area and the detection score, and the second frame is a frame at a different time on the time axis of the moving image from the first frame. [2] A system as described in [1], wherein the image detection processing associates a first detection frame or a first detection area with the image of the first frame and sets a first detection score to the first detection frame or the first detection area, associates a second detection frame or a second detection area with the image of the second frame and sets a second detection score to the second detection frame or the second detection area, and the mask area calculation unit calculates the mask area so that the mask area of ​​the detection target in the first frame is larger in an object moving situation than in an object stationary situation when the first and second detection scores are equal to or greater than a first threshold, and the object stationary situation is a situation in which the position of the detection target is the same between the first and second frames and the object moving situation is a situation in which the position of the detection target is different between the first and second frames. [3] The system according to [1] or [2], wherein the image detection process associates a first detection frame or a first detection region with the image of the first frame and sets a first detection score to the first detection frame or the first detection region, associates a second detection frame or a second detection region with the image of the second frame and sets a second detection score to the second detection frame or the second detection region, and the mask region calculation unit calculates the mask region so as not to set a mask in at least a part of a non-overlapping region when the first and second detection scores are less than a first threshold, and the non-overlapping region is the following region (a) or region (b): (a) a region inside the first or second detection frame where the first detection frame and the second detection frame do not overlap, or (b) a region inside the first or second detection region where the first detection region and the second detection region do not overlap. [4] A system according to any one of [1] to [3], wherein the image detection process associates a first detection frame or a first detection area with the image of the first frame and sets a first detection score to the first detection frame or the first detection area, associates a second detection frame or a second detection area with the image of the second frame and sets a second detection score to the second detection frame or the second detection area, and the mask area calculation unit calculates the mask area based on a figure for calculation in the image of the first frame, and the figure is determined based on the first and second detection frames or the first and second detection areas and the first and second detection scores. [5] A system described in any one of [1] to [4], wherein the image detection process associates an eye detection area for detecting eyes contained in the image with the image of each frame, and the image detection process sets the detection frame to surround the eye detection area, or sets the detection area to overlap the eye detection area. [6] [5] A system as described in [5], further comprising an eye detection unit for performing the image detection processing, wherein the eye detection unit sets an eye detection score in the eye detection area, and sets the detection score of the inner area of ​​the detection frame or the detection score of the detection area based on the eye detection score. [7] [6] The system described in [6], wherein the eye detection unit sets the detection score for the inner region of the detection frame or the detection score for the detection region according to the following rule (c) or (d): (c) When both eyes are detected for a face included in the frame image and the eye detection score equal to or greater than a second threshold is set in only one of the eye detection regions for the both eyes, the detection score for the inner region of the detection frame or the detection score for the detection region is set based on the eye detection score of the one eye detection region, and the detection frame is set to surround the one eye detection region but not the other eye detection region, and the detection regions are set to overlap the one eye detection region but not the other eye detection region. (d) When only one eye is detected for a face included in the frame image, the detection score for the inner region of the detection frame or the detection score for the detection region is set based on the eye detection score set in the eye detection region for the one eye. [8] A system according to [5] or [6], wherein the image detection process associates the eye detection area and the detection score with each of the two eyes of the face included in each of the first and second frame images, and the mask area calculation unit calculates the mask area so that a larger mask is provided on the side of the eye detection area for the left eye if the eye detection score is higher for the left eye than for the right eye in each of the first and second frame images, and the mask area calculation unit calculates the mask area so that a larger mask is provided on the side of the eye detection area for the right eye if the eye detection score is higher for the right eye than for the left eye in each of the first and second frame images. [9] A system according to any one of [1] to [8], wherein the mask area calculation unit acquires a plurality of the second frames, and the plurality of second frames include a frame that is later than the first frame and a frame that is earlier than the first frame.

[10] An image processing method that causes a computer to execute a mask area calculation step, wherein the mask area calculation step acquires a detection result of an image detection process for an image of a frame in a moving image, and the mask area calculation step calculates a mask area for an image of a first frame based on the detection result of an image of at least a second frame, and the image detection process associates a detection frame or detection area with the image of at least the second frame and sets a detection score for the detection frame or the detection area, and the detection result includes the detection frame or the detection area and the detection score, and the second frame is a frame at a different time on the time axis of the moving image from the first frame.

[11] A program that causes a computer to execute a mask area calculation process, wherein the mask area calculation process causes the computer to obtain detection results of an image detection process for an image of a frame in a video, and the mask area calculation process causes the computer to calculate a mask area for an image of a first frame based on the detection results of an image of at least a second frame, and the image detection process associates a detection frame or detection area with the image of at least the second frame and sets a detection score for the detection frame or the detection area, the detection result includes the detection frame or the detection area and the detection score, and the second frame is a frame at a different time on the time axis of the video from the first frame.

[0007] The present invention has the advantage that the mask region of the first frame is calculated appropriately. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overview of an image processing system 1 according to a first embodiment. [Figure 2] Fig. 2A is a block diagram showing the hardware configuration of the image processing system 1 according to the first embodiment. Fig. 2B is a block diagram showing the functional configuration of the control unit 11 according to the first embodiment. Fig. 2C is a block diagram showing the functional configuration of the storage unit 12 according to the first embodiment. [Figure 3]1 is an example of a processing flow of the image processing system 1 according to the first embodiment. [Figure 4] 10A to 10C are schematic diagrams illustrating an example of operation of the first embodiment (detection target P1 is hidden), and explaining an example of each process from detection to masking. [Figure 5] Fig. 5A shows an example of overlapping of the first detection frame Bk and the second detection frame Bk-n, Bk+m in one first frame fk and two second frames fk-n, fk+m (e.g., n = m = 5). Fig. 5B shows an example of a calculation figure Dk (see hatched area) in the example of Fig. 5A. Fig. 5C shows an example of the final calculation result of the mask area Mk in the example of Fig. 5B. [Figure 6] This is an example of the operation of the comparative example, in which whether or not each detection frame is masked is determined by simply comparing the detection score with the first threshold Ath1 in the images of each frame fk and fk+m. [Figure 7] Fig. 7A shows an example of overlapping of first detection frame Bk and second detection frame Bk-n, Bk-n+1, ​​Bk+m-1, and Bk+m in one first frame fk and four second frames fk-n, fk-n+1, ​​fk+m-1, and fk+m (e.g., n = m = 5). Fig. 7B shows an example of a calculation figure Dk (see hatched area) in the example of Fig. 7A. Fig. 7C shows an example of the calculation result of the final mask area Mk in the example of Fig. 7B. [Figure 8] Fig. 8A shows an example in which the detection scores in the first and second detection frames are lower than the first threshold Ath1. Fig. 8B shows an example of a calculation figure Dk (see hatched area) in the example of Fig. 8A. Fig. 8C shows an example of the calculation result of the final mask area Mk in Fig. 8B. [Figure 9] 10A to 10C are schematic diagrams illustrating an operation example (detection target P1 is not hidden) of the first embodiment, and explaining another example of each process from detection to masking. [Figure 10] 5A to 5C are schematic diagrams illustrating an example of operation (diagonal movement) in the first embodiment. [Figure 11] FIG. 10 is a block diagram showing the functional configuration of a control unit 11 according to a second embodiment. [Figure 12] 10 is an example of an image detection process (a detection frame setting method) according to the second embodiment. [Figure 13] 10 is an example of setting a detection score based on an eye detection score according to the second embodiment. [Figure 14] FIG. 10 is a schematic diagram illustrating an example of operation of the second embodiment (detection target P1a is hidden) in which the detection frames Bk, Bk-n, and Bk+m of the first and second frames fk, fk-n, and fk+m are superimposed on the image of the first frame fk. [Figure 15] 10A and 10B are diagrams illustrating an example of a detection score summing method for calculating a mask area using a detection frame according to the second embodiment. [Figure 16] Fig. 16A is an example of a figure Dk (see the hatched area) for calculation in the example of Fig. 15. Fig. 16B is an example of the calculation result of the final mask area Mk in the example of Fig. 15. [Figure 17] 10 is another example of the image detection process (detection frame setting method) according to the second embodiment. [Figure 18] 10A and 10B are schematic diagrams illustrating another example of operation of the second embodiment (detection target P1a is hidden), and explaining another example of each process from detection to masking. [Figure 19] Fig. 19A is an example of a figure Dk (see the hatched area) for calculation in the example of Fig. 18. Fig. 19B is an example of the calculation result of the final mask area Mk in the example of Fig. 18. [Figure 20] FIG. 13 is a diagram showing a modified example in which a detection frame and a detection score are set for only one eye in the case of FIG. 12. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following describes embodiments of the present invention. The various features shown in the following embodiments can be combined with each other. Furthermore, each feature can be an invention independently.

[0010] In summary, the embodiments provide a technique for performing mask processing on a moving image. In the embodiments, a detection target is detected by image detection processing, and detection results for a target frame for calculating a mask area and detection results for several frames temporally before and / or after the target frame are obtained. A mask area is calculated based on these detection results (specifically, detection frames and detection scores). First and second embodiments will be described in detail below.

[0011] 1. First Embodiment (1-1. System configuration overview) As illustrated in FIG. 1, an image processing system 1 of the first embodiment is connected to a user terminal 3 via a communication line 2. The user terminal 3 uploads a video R captured by an arbitrary imaging device (e.g., a digital video camera) to the image processing system 1. The video R is composed of a plurality of frame images. The image processing system 1 performs a masking process on the video R to generate a masked video RM in which a mask M is added. The masked video RM can be downloaded to the user terminal 3.

[0012] (1-2. Hardware configuration of image processing system 1) As illustrated in FIG. 2A, the image processing system 1 includes a control unit 11, a storage unit 12, and a communication unit 13. The image processing system 1 may also include an operation input unit 14 and a monitor 15 as shown. The operation input unit 14 is configured with a keyboard, a mouse, etc., and accepts input of various operations. The control unit 11 controls the overall operation of the image processing system 1. A portion of the storage unit 12 is configured with, for example, a random access memory (RAM) or a dynamic random access memory (DRAM). A portion of the storage unit 12 is used as a work area when the control unit 11 executes processes based on various programs. A portion of the storage unit 12 is, for example, a non-volatile memory such as a read-only memory (ROM), or various large-capacity storage devices. A portion of the storage unit 12 stores various data, programs used in processing by the control unit 11, etc. The programs stored in the storage unit 12 include, for example, an operating system (OS) for implementing basic functions of the image processing system 1, drivers for controlling various hardware, and programs for implementing various functions. The programs stored in the storage unit 12 include the computer program according to the first embodiment. The communication unit 13 has a function of connecting to the communication line 2. These components such as the control unit 11 are electrically connected to each other via a system bus.

[0013] (1-3. Functional Block Configuration of Control Unit 11 and Storage Unit 12) As illustrated in Fig. 2B, the control unit 11 includes an image data acquisition unit 11a, an image detection unit 11b, a mask region calculation unit 11c, and a mask image generation unit 11d. As illustrated in Fig. 2C, the storage unit 12 includes various databases (DBs). Specifically, the storage unit 12 includes an image DB 12a, a detection information DB 12b, a mask information DB 12c, and a masked image DB 12d.

[0014] (1-4. Processing Operation of the First Embodiment) (1-4-1. Overview of the process) The image data acquisition unit 11a receives the moving image R and stores in the image DB 12a data on a plurality of frame images that make up the moving image R. Each frame image has a frame ID, and each frame in the moving image R can be uniquely identified by the frame ID.

[0015] The following processing operations will be described with reference to Fig. 3, Fig. 4, and Fig. 5A to Fig. 5C. Fig. 3 illustrates a processing flow of the first embodiment. Fig. 4 shows a first operation example of the first embodiment (detection target P1 is hidden), depicting one scene in a video R. Figs. 5A to 5C show an example of a method for calculating a mask area.

[0016] The processing flow in FIG. 3 shows the processing of the image detection unit 11b, the mask area calculation unit 11c, and the mask image generation unit 11d. The mask area calculation unit 11c receives the processing result of the image detection unit 11b, calculates a mask area for each frame image, and transmits the calculated mask area to the mask image generation unit 11d. The image detection processing of the image detection unit 11b is not limited, and any of various known image detection technologies can be adopted, such as object detection or area detection (e.g., image segmentation). The image detection unit 11b may perform, for example, rule-based image detection processing, or may include, for example, an image detection model that performs machine inference. The image detection model may be a model that implements learning parameters obtained by performing any machine learning such as a deep learning algorithm. In the first embodiment, as an example, the image detection unit 11b employs an object detection model that performs machine inference. As an example, the image detection unit 11b can employ any of various known detection models depending on the application and required accuracy. For example, in the first embodiment, any machine-learned detection model for detecting human faces can be adopted, and the detection model may be, for example, a neural network in which trained parameters based on a deep learning algorithm are implemented.

[0017] The mask area calculation unit 11c acquires the detection result of the image of a frame in the video R in each mask area calculation process. The image of the frame is associated with the detection result (for example, a detection frame and a detection score) of the detection target detected by the image detection unit 11b. In the first embodiment (and the second embodiment described below), frames in the video are classified into a "first frame" and a "second frame." The first frame is a frame for which the mask area is to be calculated. The second frame is one or more frames at a different time point from the first frame on the time axis of the video. The second frame specifically includes a frame that is earlier and / or later than the first frame.

[0018] Next, to give an overview of Fig. 4, the four screens in the left column of Fig. 4 show the detection results of the image detection process superimposed on the image of each frame. The two screens in the center column of Fig. 4 are diagrams for explaining the process of the mask area calculation unit 11c, and show the detection frames and detection scores of the first and second frames superimposed on the image of the first frame. The two screens in the right column of Fig. 4 show the calculated mask area M k , M k+m In Figure 4, frame f k-n In the frame f, a part of the detection target P1 is hidden behind the non-detection target P2. In FIG. 4, as an example, the detection target P1 is a human face, and the non-detection target P2 is the back surface of the monitor. k Thereafter, the detection target P1 is gradually exposed.

[0019] In the example in Figure 4, frame f k-n is the frame f k It is a frame that goes back about n frames before frame f. k+m is the frame f k The frame is m frames ahead of the current frame. n and m are any natural numbers, and n=m=5 may be used as an example.

[0020] (1-4-2. Steps S100 to S102 in Figure 3) 3, first, in step S100, the image detection unit 11b acquires frame images of the moving image R from the image DB 12a. Next, the image detection unit 11b performs image detection processing on each acquired frame image (S101).

[0021] The image detection unit 11b performs image recognition and object detection on each frame image and outputs a detection result. The detection result includes a detection frame that identifies the detection object and a detection score for the detection frame. In step S102, the image detection unit 11b registers the detection results for all frame images in the detection information DB 12b.

[0022] Referring to FIG. 4, the image detection unit 11b detects frame f k-n , f k , f k+m , f k+2m The image detection unit 11b performs image detection processing on each of the frames. k-n , B k , B k+m , B k+2m and output the detection scores (0.2, 0.3, 0.8, 0.9). The detection results are stored in the detection information DB 12b.

[0023] (1-4-3. Steps S200 to S205 in Figure 3) Following step S102, the mask region calculation unit 11c executes the processes of steps S200 to S205. In this example, the mask region calculation unit 11c applies the processes of steps S200 to S205 to all frame images of the moving image R. That is, the processes of steps S200 to S203 are repeated while sequentially setting each frame f1, f2, f3, ... of the moving image R as the first frame.

[0024] For example, frame f in Fig. 4 k is set as the first frame (where k is a natural number greater than 5). In this case, the mask region calculation unit 11c calculates the first frame f k and a plurality of second frames f k-n , f k+mThe detection result of frame f in the left column of Figure 4 is obtained (S200, S201). k , f k-n , f k+m Please refer to the screen of the first frame f k In the image, the first detection frame B for detecting the detection target P1 is k The second frame f k-n , f k+m In the image, the second detection frame B k-n , B k+m and the second detection score (0.2, 0.8). These detection results are k The mask area is calculated using the

[0025] Next, the mask area calculation unit 11c calculates the first frame f k mask region M for the image k at least the second frame f k-n , f k+m In the first embodiment, for example, the first frame f k and the second frame f k-n , f k+m The detection results for each image (i.e., the first detection frame B k and second detection frame B k-n , B k+m The process of step S202 is performed based on the frame f k If you look at the screen in the row, you will see that the first detection frame B k and second detection frame B k-n , B k+m The first detection score (0.3) and the second detection score (0.2, 0.8) are overlaid.

[0026] Here, a specific example of the process in step S202 will be described with reference to FIGS. 5A and 5B. k The mask area M k The figure shows an example of the procedure for calculating the three detection frames B k-n , B k , B k+nAs an example, the mask area calculation unit 11c adds a detection score in an overlapping area where the detection frames overlap. For example, the area A2 is the first detection frame B. k (First detection score 0.3) and second detection frame B k-n Since the area A2 is an overlapping area where the first threshold A (second detection score 0.2) overlaps the area A1, the total score is 0.5. The total scores of each overlapping area are shown on the right side of FIG. 5A. In FIG. 5B, the total score is calculated by multiplying the total score by the first threshold A th1 The above areas A2, A3, A4, A5 and A7 are selected as candidates for the mask area. th1 For example, the value is 0.5. The figure defined by the perimeter of these selected areas (A2, A3, A4, A5, and A7) is the "figure for calculation D" for calculating the mask area. k " Figure D k are the first and second detection frames B k-n , B k , B k+n and can be determined based on the first and second detection scores. k is actually the frame f k The mask area M is a set of pixels in the image and can be specified by pixel coordinate data in the image. k is shape D k As an example, the mask area calculation unit 11c may calculate the mask area based on the figure D k The bounding rectangle of the first frame f k The final mask region M k (See Figure 5C)

[0027] The mask area M calculated in step S202 k is stored in the mask information DB 12c (S203).

[0028] Thereafter, it is determined whether the mask area calculation process has been completed for all frames (S204). For example, it may be determined whether the total number of frames in the moving image R matches the frame number k of the current first frame. If the process is not completed, the process of the mask area calculation unit 11c shifts the first frame by one (i.e., increments k) and then returns to step S200 (S205). In this way, the frame f k , f k+1 , f k+2 are sequentially set in the first frame, and the processes of S200 to S203 are applied to all frames.

[0029] Referring to Figure 4, frame f k+m In the center column of Fig. 4, the first frame f k+m As you can see on the screen in the row, the first frame f k+m To calculate the mask area, the first frame f k+m and the second frame f k , f k+2m Detection frame B k , B k+m , B k+2m and detection scores (0.3, 0.8, 0.9) are obtained (S200, S201). k+m and the second frame f k , f k+2m Detection frame B for each image k+m , B k , B k+2m and based on their detection scores, the first frame f k+m mask region M for the image k+m (S202). The calculated mask area M k+m The result is stored in the mask information DB 12c (S203). After that, the process proceeds to step S204 again. The processes of S200 to S205 are looped until the mask area calculation process is completed for all frames.

[0030] When the mask area calculation process is completed for all frames, the determination result of step S204 becomes affirmative (YES), and the process then proceeds to step S301.

[0031] In this way, the mask area calculation unit 11c sequentially sets each frame in the moving image R as the first frame. This allows the mask area calculation unit 11c to perform the mask area calculation process on all frame images of the moving image R. Note that the first frame may be one of several frames f near the beginning of the moving image R. k and some frames f near the end of video R k Depending on the values ​​of n and m, the second frame f k-n , f k+m In such a case, either a future or past frame may be used as the second frame to calculate the mask region.

[0032] (1-4-4. Steps S301 to S302 in Figure 3) The mask image generation unit 11d performs a predetermined masking process on each frame image in accordance with the calculated mask area (S301). The masking process may be, for example, a mosaic process. The masked image is registered in the masked image DB (S302). This generates a masked video RM, which can be downloaded to the user terminal 3.

[0033] According to the first embodiment described above, it is possible to appropriately calculate the mask area of ​​the first frame. That is, in the image detection process, the detection result includes a detection score. The detection score indicates the accuracy (likelihood) of detecting the detection target. According to the first embodiment, it is possible to include the detection result of the second frame (i.e., the first detection frame and the first detection score) in the process of the mask area calculation unit 11c. This makes it possible to appropriately calculate the mask area of ​​the first frame.

[0034] The detection score obtained by the image detection process S101 is not always stable, but changes in various ways due to, for example, the movement of the detection target P1 in the video R. For example, within the video R, the detection target P1 may be displaced (specifically, for example, translation and / or rotation around an arbitrary rotation axis), the detection target P1 may be hidden by another object (non-detection target P2), whiteout may occur in the image, or a detection error may occur due to the detection model of the image detection unit 11b. In these cases, image detection may fail, or even if image detection is successful, the detection score may be unstable. The first embodiment has the advantage of being able to appropriately calculate the mask area even under such circumstances.

[0035] (1-5. Comparative Example) In the comparison example in Figure 6, the detection score and the first threshold A th1 By simply comparing the frame f with the frame f, the presence or absence of a mask is determined. k When the detection score is low, the mask area is not calculated. In this regard, according to the first embodiment, the mask area can be calculated appropriately (see FIG. 4).

[0036] (1-6. Modifications, etc.) (1-6-1. Variations of image detection processing and mask area calculation) 7A to 7C show one first frame f k and four second frames f k-n , f k-n+1 , f k+m-1 , f k+m In the first detection frame B k and second detection frame B k-n , B k-n+1 , B k+m-1 , B k+m Even if the number of second frames increases, the calculation method illustrated in FIGS. 5A to 5C can be similarly applied. However, in the example of FIGS. 7A to 7C, the first threshold A th1 Average score A ave Let's say. A ave The first and second detection frames B k , B k-n , B k-n+1 , B k+m-1, B k+m In the example of FIG. 7A, the total value of the first and second detection scores is 1.9, and the number of first and second detection frames is 5. ave is 1.9 / 5=0.38. th1 Using the same method as in Figure 5B, we obtain the shape D k may be calculated, and the figure D k The circumscribing rectangle of the mask area M k It may be calculated as:

[0037] In the example of FIGS. 8A to 8C, detection frame B k+m In the example of FIGS. 8A to 8C, the first detection frame B k The first detection score (0.3) and the second detection frame B k-n , B k+m Both of the second detection scores (0.2, 0.4) are below the first threshold A th1 is less than (here A th In such a case, the mask area calculation unit 11c calculates the first frame f so as not to set a mask on at least a part of the non-overlapping areas A1, A6, and A7. k The mask area M k The non-overlapping regions A1, A6, and A7 may be calculated as follows: k , B k-n , B k+m Inside the first detection frame B k and second detection frame B k-n , B k+m The non-overlapping areas A1, A6, and A7 are areas where the first and second detection frames B k , B k-n , B k+m The non-overlapping areas A1, A6, and A7 have low detection scores and do not overlap with each other. Since the non-overlapping areas A1, A6, and A7 have low mask priorities, at least a part of the non-overlapping areas A1, A6, and A7 may be left unmasked. This allows the mask area M k It is possible to prevent excessive calculation of

[0038] In addition, in FIGS. 8A to 8C, the mask area M k is shape D k Therefore, the entire non-overlapping area A6 and parts of the non-overlapping areas A1 and A7 are included in the mask area M. k However, this is just an example, for example, in the mask region M k Shape D k The inscribed rectangle can be determined as follows. k The inscribed rectangle with the largest area may be used. In the example of FIG. 8A, this largest inscribed rectangle is a rectangle consisting of height h4 and width W2. Height h4 is the height of area A4. Width W2 is the total width of areas A2, A3, and A4. This largest inscribed rectangle of h4 × W2 is used as the mask area M. k In this case, all of the non-overlapping areas A1, A6, and A7 are not masked.

[0039] (1-6-2. Other operation examples) FIG. 9 illustrates a different scene from FIG. 4 in the video R. In the example of FIG. 9, the detection target P1 is closer to the imaging device (digital video camera) than the non-detection target P2, and the entire detection target P1 is always detected in each frame. k-n , f k , f k+m , f k+2m In the example of Figure 9, the detection scores for each detection frame (0.7, 0.8, 0.8, 0.9) are all above the first threshold A. th1 That's all (in this example, A th =0.5). However, the position and size of the detection frame change as the detection target P1 moves. In such a case, the mask area is calculated as follows by the processing steps S200 to S205 (see FIGS. 3, 4, and 5A to 5C) of the first embodiment described above.

[0040] As an example, frame f k The mask region calculation unit 11c calculates the first and second detection scores (0.7, 0.8, 0.8) by the first threshold A th1If this is the case, the first frame f k The mask area M of the detection target P1 in k The mask area M k Here, the "object stationary state" refers to the first and second frames f k , f k-n , f k+m The "object moving state" refers to a state in which the position of the detection target P1 is the same between the first and second frames f k , f k-n , f k+m 9 shows an example of a situation in which the position of the detection target P1 is different between the two.

[0041] If the detection target P1 in FIG. 9 is k At the position of the first and second frames f k , f k-n , f k+m In such a situation where the object is stationary, unless there is a special situation such as a detection error or overexposure, the detection frame will be the first detection frame B. k The detection score is also substantially unchanged from 0.8. Therefore, the mask area M k is the first detection frame B k In contrast to this, in the object moving situation illustrated in FIG. 9, the first detection frame B k The second detection frame B is located outside of k-n , B k+m is generated, and as a result, the first detection frame B k The mask area M extends to the outside of k In this way, the mask area M k may be calculated for a larger area.

[0042] (1-6-3. Other operation examples ii: Tilt detection by image detection unit) As shown in FIG. 10, the detection target P1 may move obliquely within the screen. When the detection target is tilted within the images of the first and second frames, the image detection unit 11b detects an angle corresponding to the tilt (angle θ in FIG. 10).k ) may be generated so that the detection frames of the first and second frames are tilted (detection frame B in FIG. 10). k-n , B k , B k+m The mask area calculation unit 11c may tilt the mask area of ​​the image of the first frame in accordance with the tilt of the first and second detection frames (mask area M in FIG. 10). k ) As an example, in FIG. 10, the first frame f k First detection frame B k and the second frame f k-n , f k+m Second detection frame B k-n , B k+m is the angle θ k In this case, the first and second detection frames B k , B k-n , B k+m The mask area M circumscribing k also has the same angle θ k The mask area M may be set at an angle. k may extend diagonally across the image. k , f k-n , f k+m If there is an error in the tilt angle between, for example, the mask area M k In the example of FIG. 10, the first frame f k Angle θ at k The mask area M is set as close as possible to the detection target P1 that moves diagonally across the screen in the video. k It becomes possible to calculate

[0043] (1-6-4. Processing based only on the detection result of the second frame) In the first embodiment, the calculation process (S200 to S205, see FIGS. 3 to 5C, and 7A to 10) using the detection results of both the first and second frames has been described as an example, but this is merely an example. As another example, the detection result of the first frame may not be used (i.e., S200 in FIG. 3 is omitted), and the mask area calculation process of step S202 may be performed based on the detection result of only the second frame (detection frame and detection score). As yet another example, when there are detection results of both the first and second frames, processing may be performed based on both detection results. On the other hand, when there is no detection result in the first frame, the mask area may be calculated based on the detection result of only the second frame (detection frame and detection score). For example, a case-by-case process may be performed in step S202. An example of a situation in which these processes can be used is, for example, if the first frame f k When whiteout or the like occurs in the first frame f k There is a possibility that the image recognition of the detection target P1 in the image of the first frame will fail. As another example, it is possible to imagine a case where an image detection error (failure in face recognition) occurs, or a case where "when a detection score equal to or greater than a certain threshold and a detection frame associated with it are adopted, the detection score in the first frame is less than the threshold." In such a case, the first frame f k In any scene, the detection window and the detection score may not exist in the image of the second frame f k-n , f k+m By using the detection result (including the detection score), the mask area calculation unit 11c calculates a mask area M in the vicinity of the detection target P1. k This modified example can also be adopted in the second embodiment described later.

[0044] 2. Second Embodiment In the second embodiment, differences from the first embodiment will be mainly described, and descriptions of matters common to or similar to the first embodiment will be omitted or simplified.

[0045] (2-1. Functional block configuration of the control unit 11) 11A and 11B, the control unit 11 of the second embodiment includes an image detection unit 111b. This is provided in place of the image detection unit 11b of the first embodiment. In this respect, the second embodiment differs from the first embodiment. The image detection unit 111b includes a face image detection unit 111b1 and an eye detection unit 111b2. As with the image detection unit 11b, the face image detection unit 111b1 and the eye detection unit 111b2 can also employ various known image detection techniques, and therefore detailed description thereof will be omitted.

[0046] (2-2. Processing Operation of Second Embodiment) The processing flow of the second embodiment has much in common with the processing flow of the first embodiment (see FIG. 3), but there are some differences in the image detection process S101 and the mask area calculation process S202. The rest of the processing is basically the same as or can be diverted to the first embodiment, so the explanation will be omitted or simplified as necessary.

[0047] (2-2-1. Overview of eye detection area and eye detection score) In the second embodiment, the image detection unit 111b first executes the same processes as S100 to S102 in FIG. 3. The image detection process (corresponding to S101) according to the second embodiment will be described with reference to FIG. 12 and other figures. In FIG. 12, as an example, the image detection process for frame f k-n The image to be processed is

[0048] In the image detection process (S101), the eye detection unit 111b2 detects frame f k-n Detect the eye (see detection target P1a) in the image and set the eye detection area C k-n The frame k-n In the second embodiment, the eye detection area C k-n Detection frame B surrounds k-n As an example, the eye detection area C k-n may be set as a point (a point in image coordinates) in the image. k-n The eye detection area C may cover part or all of each eye in the image. k-nAt least one eye is set. If the face is facing almost forward, two eyes are usually detected unless the face is hidden by an obstacle. If the face is facing diagonally or hidden by an obstacle, one eye may be detected. In the example of FIG. 12, the eye detection unit 1112b detects one detection frame B k-n However, there are two eye detection areas C k-n is set to surround both.

[0049] In the second embodiment, as an example, as in step S210 of FIG. 12, the face image detection unit 111b1 first detects a face in an image and generates a detection frame B' k-n Then, the eye detection unit 111b2 sets the detection frame B' k-n The eye detection unit 111b2 detects the eye (i.e., the detection target P1a) inside the eye detection area C k-n Next, in step S211, the eye detection unit 111b2 sets the two eye detection areas C k-n A horizontal axis HL is defined by connecting the centers of the two horizontal lines. Furthermore, two horizontal lines are set vertically, starting from the horizontal axis HL, at a predetermined distance apart (see the vertical bidirectional arrows in S211). In the second embodiment, as an example, the two horizontal lines and the face detection frame B' k-n The rectangle surrounded by and is called "detection frame B k-n As another example, the inner area of ​​the rectangle is treated as "detection area B k-n It is possible to treat it as " ". Note that performing eye detection after face detection has the advantage of higher accuracy than performing eye detection directly. However, it is not limited to this. k-n The eye detection area may be set directly without performing the above steps (setting of the eye detection area).

[0050] (2-2-2. First Example: Setting detection scores by interpolation, etc., and calculating mask areas) In the first example, the eye detection unit 111b2 detects each eye detection area C k-n Based on the detection score (0.3, 0.7) of detection window B k-n12. The eye detection unit 111b2 sets the detection score for the detection frame B. k-n The first end of the horizontal direction is assigned a detection score of 0.3. k-n The eye detection unit 111b2 is the end closest to the detection frame B k-n The second end of the horizontal direction is assigned a detection score of 0.7. k-n In the first example, the eye detection area C k-n Detection scores are set at both ends of the detection frame according to the positional relationship with the object.

[0051] Next, please refer to FIG. 13. The eye detection unit 111b2 detects the eye in the detection frame B k-n In the inner region of the detection frame B, the detection score is linearly interpolated from 0.3 at the first end to 0.7 at the second end. k-n The detection score is set in a gradient (so to speak, in a gradation) in the inner region of the detection frame B. The graph at the bottom of FIG. 13 illustrates the distribution of the detection scores when the horizontal coordinate is conveniently expressed as numbers from 0 to 10. By doing so, the eye detection unit 111b2 sets the detection frame B based on the eye detection scores of 0.3 and 0.7. k-n The eye detection unit 111b2 sets a detection score for the inner area of ​​the detection frame B. k-n The inner area of ​​the eye detection area C k-n It is possible to set a detection score according to the positional relationship with the other detection frame B. k , B k+m The detection score for each inner region can also be set.

[0052] Next, a first example of the processing (corresponding to S200 to S205 in FIG. 3) of the mask region calculation unit 11c of the second embodiment will be described with reference to FIG. 14. In FIG. 14, a frame f k is the first frame, and frame f k-n , f k+m The mask area calculation unit 11c calculates the first and second frames f in the moving image R in one mask area calculation process. k , fk-n , f k+m The detection results of the image detection process for each image in the first detection frame B are obtained. k The inner area and detection score (linear interpolation between 0.5 and 0.8) of the second detection frame B k-n The inner area and detection score (linear interpolation between 0.3 and 0.7) of the second detection frame B k+m and a detection score (linearly interpolated between 0.8 and 0.9).

[0053] Next, please refer to Figure 15. The top part of Figure 15 shows the first frame f k The first and second detection frames B are superimposed on the image of k , B k-n , B k+m The graph in the middle of FIG. 15 shows the first and second detection frames B k , B k-n , B k+m The lower part of Fig. 15 shows the distribution of detection scores (both linearly interpolated) in the first and second detection frames B. k , B k-n , B k+m 15 shows an example of the result of adding up the detection scores when the detection frames are overlapped. As an example, the score addition may be performed by simply adding up the scores in the inner areas of the detection frames according to the overlap of the detection frames. The detection score distributions along the XX line, YY line, and ZZ line in the upper part of FIG. 15 are listed in the graph in the lower part of FIG. 15.

[0054] The upper part of FIG. 16A shows the figure D in the first example. k The graph in the lower part of FIG. 16A shows an example of the first threshold A th1 (In this example, A th = 0.5). For example, the mask area calculation unit 11c calculates a th The above area is represented by figure D k Furthermore, as shown in FIG. 16B, for example, the mask area calculation unit 11c calculates the mask area of ​​the figure D k The circumscribing rectangle of the mask area M k Other shapes D k and the mask region M kFor details of the processing, please refer to the first embodiment (S202, explanations of FIGS. 4 and 5, etc.). The processing after calculating the mask area is the same as in the first embodiment (see FIG. 3, etc.), so the explanation will be omitted.

[0055] In the first example, the detection scores are set at both ends of the detection frame based on the eye detection score. However, this is only an example. For example, a detection score according to the value of the eye detection score may be set at the horizontal coordinate position corresponding to the eye detection area. For example, in the detection frame B k-n Eye detection area C with a low score of 0.3 k-n is located at coordinate "3" and has a high score of 0.7. k-n is assumed to be located at coordinate "7." In this case, in a graph (see FIG. 13 or FIG. 15) with coordinates as the first axis and detection score as the second axis, a detection score of 0.3 may be set at the position of coordinate 3, a detection score of 0.7 may be set at the position of coordinate 7, and a function (for example, a linear function) passing through these two points may be calculated. The detection score that changes according to this function may be expressed as a detection frame B k-n In this case, the detection score is calculated by interpolation for the area inside the two points, and by extrapolation for the area outside the two points.

[0056] There are also various variations in calculation methods, such as interpolation. For example, in a first example, a detection score that changes according to a linear function depending on the horizontal coordinate is set in the inner region of the detection frame by linearly interpolating the detection scores at both ends of the detection frame. Here, the horizontal direction refers to the longitudinal direction of the detection frame, that is, the direction in which the two eyes are lined up. However, this is merely an example. The detection score may be changed, for example, in a stepped or curved manner, or may be changed continuously or discontinuously, depending on the horizontal coordinate of the detection frame. The detection score distribution may vary monotonically (i.e., monotonically increase or decrease) from one end of the detection frame to the other, but is not limited to this. For example, a detection score distribution may be adopted that has a maximum value at the coordinate position of the eye detection region and decreases in the surrounding area.

[0057] (2-2-3. Second Example: Detection Score Setting by Area Division and Mask Area Calculation) A second example will be described with reference to FIGS. 17 to 19B. As a second example, when the eye detection unit 111b2 detects the detection frame B k-n The detection frame B may be divided into multiple regions (see S222 in FIG. 17). k-n may be divided into multiple regions, and each region may be associated with a different detection score.

[0058] Specifically, the eye detection unit 111b2 detects the eye in the detection frame B k-n Each eye detection area C k-n The edge region including the eye detection region C k-n One of the edge regions (the left side of the paper in FIG. 18) is the eye detection region C with an eye detection score of 0.3. k-n Therefore, one of the edge regions (left side of the paper in FIG. 18) is associated with a detection score based on an eye detection score of 0.3 (0.3 is used as is in this example). Similarly, the other edge region (right side of the paper in FIG. 18) is associated with a detection score based on an eye detection score of 0.7 (0.7 is used as is in this example). The middle region is composed of two eye detection regions C k-n As an example, the average value of the edge region detection scores (i.e., 0.5) is set to correspond to the edge region detection score.

[0059] Such processing by the eye detection unit 111b2 may be performed on all frame images of the moving image R. The detection results are stored in the detection information DB 12b (see step S102 in FIG. 3).

[0060] Next, a second example of the process (corresponding to S200 to S205 in FIG. 3) of the mask region calculation unit 11c of the second embodiment will be described with reference to FIG. 18. In FIG. 18, as in FIG. 14, k is the first frame, and frame f k-n , f k+m In the second example, the mask region calculation unit 11c also calculates the first and second frames f k , f k-n , f k+m The detection results of the image detection process for each image in the first detection frame B are obtained.k Each region and detection score (0.5, 0.65, 0.8) and the second detection frame B k-n Each region and detection score (0.3, 0.5, 0.7) and the second detection frame B k+m The mask region calculation unit 11c calculates the first and second detection frames B having a plurality of regions and the detection scores (0.8, 0.85, 0.9). k , B k-n , B k+m Based on the detection score set for each region, the figure D k (See FIG. 19A) and the mask area M k (See Figure 19B) k and the mask region M k can be calculated in the same way as in the first embodiment (see the explanations of S202, FIGS. 4 and 5, etc.).

[0061] According to the second example, the eye detection unit 111b2 detects the first and second frames f k , f k-n , f k+m The first and second detection frames B are placed in each image. k , B k-n , B k+m and the first and second detection frames B k , B k-n , B k+m Each inner region is divided into multiple regions (edge ​​regions and middle regions), and the detection score of the edge region including the eye detection score is set based on the eye detection score, and the eye detection region C k Based on the detection score of the edge region including the eye detection region C k It is possible to set a detection score for an intermediate region that does not include the eye. Since it is possible to take into account the score difference between eye detection regions of the same face detection frame, it is possible to accurately calculate a mask region for masking the eyes.

[0062] In the second example, the number of divisions into which the inner region of the detection frame is divided is not limited. The intermediate region may be divided into any number of regions, or multiple intermediate regions may be set. The edge region may be divided, or a first edge region including the eye detection region and a second edge region outside the first edge region that does not include the eye detection region may be set, or the second edge region may be further divided. The number of divisions into the intermediate region and / or the second edge region may be increased. Specifically, the number of divisions into the intermediate region may be, for example, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, or 20, or may be within a range between any two of the numbers exemplified here. This number of divisions can also be adopted for the second edge region. Depending on the resolution, the width of each divided region may be very small, somewhat close to the pixel width (for example, 10 times or less, 50 times or less, 100 times or less, or 200 times or less of the pixel width). As an example, when the intermediate region is divided into multiple regions, the detection score of each region may be changed in a stepwise manner from one end to the other end of the intermediate region. This stepwise change may be such that the eye detection score of the eye detection region on one end side is the starting value and the eye detection score of the eye detection region on the other end side is the final value. A monotonic change may also be made between the two eye detection scores. The detection score of each region when the second end region is divided into multiple regions may also be increased or decreased in a stepwise manner as the region is further away from the eye detection region.

[0063] (2-2-4. Summary of processing operations) As described above, the image processing system 1 of the second embodiment further includes the eye detection unit 111b2 for executing image detection processing. k , C k-n , C k+m The eye detection unit 111b2 sets the eye detection score to the detection frame B k , B k-n , B k+m The detection score is set for the inner area of ​​the eye detection area C (see FIG. 13 or FIG. 17). k , C k-n , C k+mSpecifically, at least the first example (interpolation in FIG. 13, etc.) and the second example (area division in FIG. 17) may be adopted.

[0064] One of the features of the processing operation of the second embodiment is as follows. In the second embodiment, the first and second frames f k , f k-n , f k+m Both eyes are detected in the faces included in all images of the frame f. For example, in the frame f in Fig. 14, the eye detection area and the detection score are set for each eye. k-n The eye detection scores for the right eye and left eye are 0.3 and 0.7, respectively, and the eye detection scores for the eye detection regions associated with the left eye are higher than those for the right eye. k , f k+m In this case, the mask area calculation unit 11c calculates the first frame f so that a larger mask is generated on the side of the left eye detection area than on the side of the right eye detection area. k The mask area M k (See FIG. 16B). Conversely, if the detection score of the right eye is higher than that of the other two eyes, a larger mask can be generated on the right eye side. The "side of the eye detection area with the higher detection score" can be defined, for example, as follows: k The perpendicular bisector of the imaginary line connecting these is the imaginary central axis CL k (See Fig. 16A, Fig. 18 and Fig. 19B.) The virtual central axis CL k The eye detection area C with a high score is k The side of the virtual central axis CL may be defined as "the side of the eye detection area with the higher eye detection score." k The eye detection area C on the low score side is the boundary. k Eye detection area C on the higher score side k The mask area M k may be calculated.

[0065] (2-3. Modifications of the Second Embodiment) The method of setting the detection frame so as to surround the eyes is one example, and as another example, a detection frame with a predetermined aspect ratio that surrounds the eye detection region may be set so as to cover the eye detection region. k may be omitted (that is, only the eye detection process may be performed without performing face image detection), and in this case too, the detection frame may be set based on the eye detection region.

[0066] For example, there may be cases where a profile is to be detected, or where one eye is covered by a hand or the like and is not captured in the video. The eye detection unit 111b2 detects each detection frame B in accordance with the following rule (c) or (d): k-n , B k , B k+m First, we will explain rule (c). In the example of Figure 12, frame f k-n In the image, both eyes (two detection targets P1a) are detected, and the eye detection area C for one eye (left eye) k-n Only the second threshold A th2 The eye detection score (0.7) is set (for example, A th2 = 0.5). In such a case, as rule (c), detection frame B k-n is the eye detection area C of one eye (left eye). k-n and the eye detection area C of the other eye (right eye). k-n The detection frame B does not need to be enclosed. k-n The detection score of the face detection frame B' in FIG. 12 may be set to 0.7 (see FIG. 20). k-n In case of a detection error or eye detection area C k-n If the detection score is very low, it may happen that the eye detection region for one eye (for example, the right eye) is not generated. In such a case, as rule (d), the eye detection region for one eye is generated by the detection frame B k-n In the inner region of the detection frame, a detection score of 0.7 may be set in the same manner as above (see FIG. 20). Note that a predetermined increase or decrease correction may be applied to 0.7. In rule (c) or (d), the detection score may be uniform in the inner region of the detection frame. th2 is the first threshold A th1 It may be the same as

[0067] The second embodiment is not limited to these and can be modified in various ways. The various features described in the modifications of the first embodiment can be adopted in the second embodiment (see, for example, "1-6. Modifications, etc." of the first embodiment).

[0068] <3. Other modifications, etc.> (3-1. Variations of image detection processing) The image detection process of the image detection unit 11b does not have to be object detection processing, but may also be region detection processing. Any known technology can be used for the region detection process, such as a rule-based region detection process, or machine inference (e.g., so-called image segmentation) using a trained region detection model. Some region detection processes can classify image pixels according to the object class to which they belong. For example, in region detection processing, a set of pixels of the detection target (e.g., a face) may be provided as the detection region. For example, a group of pixels with a detection score above a certain threshold may be formed, and a process such as filling missing regions may be performed on this group of pixels to set the detection region. The outer boundary of the detection region (i.e., the boundary line defining the inside and outside of the detection region) can be treated in the same way as a detection frame. The "detection score of the detection region" can be determined using any calculation method, and may be determined, for example, based on the detection score of each pixel included in the detection region. For example, a group of pixels having a detection score equal to or greater than a certain threshold may be defined, and a "statistic" may be calculated from the detection scores of all pixels included in the group of pixels, and the detection score of the detection region may be set based on this statistical value. This statistical value may be, for example, the average, median, or mode based on the detection scores of each pixel.

[0069] As described above, the image detection unit 11b may be caused to perform the region detection process, and the detection results (detection region and detection score) of the region detection process may be replaced with the detection results (detection frame and detection score) described in the first and second embodiments and their respective modified examples. As a modified example, for example, a first detection region may be associated with the image of the first frame instead of the first detection frame, and a first detection score may be set for the first detection region. For example, a second detection region may be associated with the image of the second frame instead of the second detection frame, and a second detection score may be set for the second detection region. For example, the non-overlapping region exemplified in FIG. 8C may be, as a modified example, an area inside the first or second detection region where the first detection region and the second detection region do not overlap. Detection frame B k-n Instead of, for example, eye detection area C k-n The detection area may be set so as to overlap with the detection frame B. k-n A detection area such as a rectangular area may be set so as to occupy a position and range similar to that of the rule (c) or (d) for detecting one eye (example in FIG. 20).

[0070] (3-2. Use, detection target, etc.) In the image processing system 1 according to the first and second embodiments, there are no limitations on the content of the video R and the masking target. The video R may be, for example, a medical video (e.g., a surgery video, a treatment video, a medical examination video, or a diagnosis video), a commercial video, an industrial video (a factory video, a manufacturing process video), a streaming distribution via the Internet or the like, a security camera video, an internal or external meeting video, an interview video, or the like.

[0071] (3-2-1. Application to streaming videos, etc.) The image processing system 1 of the first and second embodiments can generate a masked video RM by performing masking on a recorded video R. However, this is not limited to this, and the system can also be used for masking streaming videos, etc. If a frame that is later than the first frame is used as the second frame, a delay of at least m frames is required to generate the masked video RM from the video R. For this reason, the distributed video (masked video RM) requires a delay of m frames or more relative to the captured video (video R), but the delay can be kept within a desired range by not making the value of m too large. Note that variations in the values ​​of n and m and the frame rate will be described later.

[0072] (3-2-2. Detection target, mask target, etc.) In the image detection process (object detection or area detection) of the embodiment and the modified example, as an example, a human face (first embodiment) or eyes (second embodiment) is used as a mask target. However, this is not limited to this, and the detection target and mask target can be set arbitrarily. A detection frame B is set according to each detection target. k The position, size, and range of the detection target and mask target can be arbitrarily set, and this also applies to variations that employ detection regions. For example, the detection target and mask target may be images of tangible objects or images of non-tangible visible objects. Tangible objects may be, for example, a human or animal face, individual eyes, both eyes and their surroundings, the upper body, or organs, or any commercial or industrial product, such as factory equipment or manufacturing equipment. Non-tangible visible objects may be, for example, symbols, diagrams, illustrations, etc. Symbols include any character symbols, such as kana characters, kanji characters, numbers, and alphabets. Any tangible object and / or symbol that can identify personal information, confidential information, and / or classified information may be used as the detection target. In medical videos, the faces and / or names of patients, doctors, medical staff, etc. may be masked, and text information such as manufacturer names and model numbers of medical devices and pharmaceuticals may be masked.

[0073] Note that the type of mask is not limited to mosaic processing or the like, and it is sufficient that the calculated mask area can be masked. The mask can be arbitrarily set according to the application or the like, and can be any illustration image (for example, a character image such as an avatar image, or an eye illustration image).

[0074] (3-3. Type and Number of Second Frames) The second frame may be a frame that is a predetermined number (the aforementioned n or m) away from the first frame, but is not limited thereto, and may also be a frame consecutive to the first frame. Also, the predetermined numbers n and m are arbitrary natural numbers, and n and m may be the same, but are not limited thereto, and may be different, and either n > m or n < m may be used. The value of n is, for example, from 1 to 120, and may be, for example, from 1 to 60. Specifically, the value of n may be, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, or 120, or may be within the range between any two of the examples shown here. The value of m is, for example, from 1 to 120, and may be, for example, from 1 to 60. Specifically, the value of m may be, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, or 120, or may be within the range between any two of the examples shown here. The values of n and m may be set according to the frame rate.

[0075] The frame rate of the moving image R processed by the image processing system 1 is not limited. The frame rate can be set in various ways depending on the intended use of the moving image. The frame rate of the moving image R may be 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 21, 22, 23, 24, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 150, 200, or 240 fps, or may be within a range between any two of the fps values ​​exemplified here. The frame rate standard of the moving image R may be, for example, the NTSC system, the PAL system, or another system. In the NTSC system, the frame rate can be selected from 60 fps, 59.94 fps, 30 fps, 29.97 fps, and 24 fps. In the PAL system, the frame rate can be selected from 50 fps, 25 fps, and 24 fps. Frame rates of 60 fps, 24 fps, and 25 fps are commonly used in movies. The frame rate of the video R may be any one of the fps values ​​exemplified here, or may be within a range between any two of the fps values ​​exemplified here. The higher the frame rate (i.e., the more frames per second), the shorter the time interval between frames, so when the frame rate is high, the above values ​​of n or m may be set to be larger.

[0076] In the first and second embodiments, the second frame includes both a frame that is earlier than the first frame and a frame that is later than the first frame (for example, the first frame f k The second frame f k-n , f k+m ). However, this is just one example. The second frame may be either a future frame or a past frame. That is, the second frame may be only one or more frames that are earlier than the first frame, or the second frame may be only one or more frames that are later than the first frame. The "multiple frames" mentioned here may, as one example, be frames extracted discontinuously on the time axis (i.e., frames spaced apart from each other on the time axis). As another example, the second frame may include multiple consecutive frames that are earlier than the first frame and / or multiple consecutive frames that are later than the first frame.

[0077] The number of second frames is not limited. Specifically, the number of second frames may be, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, or 120, or may be within a range between any two of the numbers exemplified here. The number of second frames may be one. In this case, the second frame may be either a frame that precedes the first frame or a frame that precedes the first frame.

[0078] (3-4. Modifications of detection frames, detection areas, etc., figures for calculation, and mask area calculation, etc.) In the first and second embodiments, the following process is performed as an example: First, the control unit 11 (image detection units 11b and 111b and mask region calculation unit 11c) superimposes the first and second detection frames, and calculates the detection score and the first threshold A. th1 (See FIG. 5A, FIG. 7A, FIG. 8A, FIG. 15, or FIG. 19A.) Specifically, the control unit 11 selects an area based on, for example, a first threshold value A th1 The area having the above total score is selected as the selected area. The control unit 11 sets the perimeter figure of this selected area as figure D k (For example, D in Figs. 5B, 7B, 8B, 16A and / or 19A) k The control unit 11 further k The circumscribing rectangle of the final mask area M k (For example, M in FIG. 5C, FIG. 7C, FIG. 8C, FIG. 16B or FIG. 19B) k However, these processes are merely examples, and the present invention is not limited to these.

[0079] Figure for calculating the mask area (Figure D kThere are various variations of the calculation figure (see reference). For example, the calculation figure may be an inward or outward offset of the "outer perimeter figure of the selected area." The calculation figure may also be a closed area in which each vertex of the "outer perimeter figure of the selected area" is connected by a straight line or a curve. Even if multiple detection frames are spaced apart within the screen and do not overlap, the calculation figure may be set as follows. For example, the calculation figure may be any figure that circumscribes multiple detection frames that are spaced apart at least at one point or one side and spreads across these multiple detection frames. As another example, the calculation figure may be any figure that covers part, most, or all of multiple detection frames that are spaced apart and spreads across these multiple detection frames. Figure D k These modified examples can also be adopted when a "detection region" obtained by region detection processing is used instead of a "detection frame."

[0080] First threshold A th1 is used for region selection when determining a figure to be used for calculating a mask region. In the first and second embodiments, the first threshold A th1 The second threshold A is exemplified as 0.38 and 0.5 (see FIGS. 5B, 7B, 8B, and 19A), but is not limited to these values. th2 For example, when the detection score of one detection frame or the eye detection score of one eye detection area is set within the range of 0 to 1.0, the first threshold A th1 or second threshold A th2 The first threshold A may be, for example, 0.1 to 0.9, or may be, for example, 0.3 to 0.7. th1 or second threshold A th2 Specifically, may be, for example, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, or 0.9, or may be within a range between any two of the values ​​exemplified here.

[0081] There are various variations in the shape of the mask region. As described in the first and second embodiments and their modifications, the mask region may be any shape that circumscribes or inscribes the periphery of the mask region calculation figure. This has the advantage of allowing the mask region to have a regular shape. However, the present invention is not limited to this. For example, the mask region may be set to the same shape as the calculation figure. Alternatively, the mask region may be any shape that covers a part, a majority, or all of the calculation figure. This arbitrary shape may be any polygon (for example, any polygon having a size of at least a triangle, and is not limited to a regular polygon), a polygon in which all or some of the vertices of the calculation figure are connected by straight lines, a circle, an ellipse, or an arbitrary closed curve (a curve that passes through all or some of the vertices of the mask region calculation figure).

[0082] As described with reference to FIGS. 5A to 5C, the detection score for the overlapping region (A2 to A5) of the first and second detection frames may be calculated by adding up the detection scores of the first and second detection frames, as in the first and second embodiments. The term "summing" is not limited to simple addition and is intended to have a broader meaning, encompassing various calculation methods for calculating multiple detection scores according to various rules. For example, when the first and second detection frames have an overlapping region (e.g., A2 to A5) and a non-overlapping region (e.g., A1, A6, A7), the combined detection score for the overlapping region may be calculated to be higher than the individual detection scores for the non-overlapping regions, and any such score calculation may be employed. For example, in the overlapping region, any summary statistic (e.g., mean, maximum, median, or minimum) may be calculated from the multiple detection scores and used as the detection score for the overlapping region. Alternatively, the detection score for the overlapping region may be calculated to be relatively higher as the number of overlapping detection frames in the overlapping region and / or the overlapping area increase.

[0083] In the first embodiment, all second detection frames in the second frame are used to calculate a mask region for the image of the first frame. However, the present invention is not limited to this. As a modified example, only a plurality of second detection frames whose second detection scores are greater than or equal to the first threshold A th1Detection frames below the first threshold A may be rejected. th1 It is also possible to adopt only the second detection frame described above, and then calculate a mask region based on the first detection frame and the adopted second detection frame.

[0084] The first and second detection frames are not limited to rectangular frames, but may be any polygonal frame, or may be a circular frame, an elliptical frame, or any other closed curve frame. The polygon is not limited to a regular polygon, but may have any irregular shape.

[0085] In the second embodiment, the first and second detection frames are each divided into multiple regions, but this technical concept can also be applied to the first embodiment. However, while the division is performed based on the eye detection region in the second embodiment, the first embodiment does not have an eye detection region. The region division method can be set arbitrarily. For example, as a modification of the first embodiment, the inner region of each of the first and second detection frames may be divided into a central region of the detection frame and one or more peripheral regions surrounding the central region. In this case, the detection score value for each region may be inclined, for example, so that the central region is relatively high (for example, a detection score for image detection processing is set in the central region) and the peripheral region is relatively low (for example, the score gradually decreases toward the periphery of the detection frame).

[0086] (3-5. Speeding up processing) First and second frames f k , f k-n , f k+m An image size reduction process may be applied to each image, thereby speeding up the process of step S202. The image size reduction process may be performed at any time before step S202, or may be inserted, for example, between S100 and S101.

[0087] (3-6. Variations in hardware configuration, etc.) There is no limitation on the hardware configuration of the image processing system 1. The image processing system 1 may be provided by a single server, but is not limited to this and may be provided by multiple servers. The functional units of the control unit 11 and the storage unit 12 may be provided together on one server, or may be separately located on two or more servers. Furthermore, one functional unit may be provided on one server, but some of the processing of one functional unit may be executed by one server and the remaining processing may be executed by one or more other servers.

[0088] When a computer executes a program to perform each processing step, thereby realizing the image processing system 1 or the image processing method according to the first and second embodiments, the program may be stored in the storage unit 12 or in a non-transitory computer-readable recording medium. The non-transitory storage medium may be provided to consumers. The image processing system 1 may be realized by so-called cloud computing, by reading a program stored in an external storage device. Alternatively, when the image processing system 1 is realized by hardware, it can be realized by various circuits such as an ASIC, a SOC, an FPGA, or a DRP. Furthermore, at least some of the functional components of the image processing system 1 may be processed by software and / or hardware on a user terminal 3, etc.

[0089] <4. Features of the eye detection unit 111b2> The image detection process of eye detection unit 111b2 (corresponding to S101 in FIG. 3) included in the second embodiment can be performed independently and does not necessarily have to be used in conjunction with the process of mask area calculation unit 11c (S200 to S205). The process of eye detection unit 111b2 can constitute an invention on its own, and is a technical feature that can be filed as a divisional application.

[0090] For example, first, one detection frame B as shown in Figure 13 k-n 17. In step S222 of FIG. 17, the detection score is set in the inner area of ​​the detection frame B by interpolation or the like. k-nThen, the detection frame B is divided into three regions. k-n The detection score of each region and the first threshold A th1 and based on the comparison result, a first threshold A th1 The above-mentioned region may be calculated as a mask region. In other words, the image processing exemplified in Fig. 14 or 18 (i.e., image processing using multiple detection frames based on the first and second frames) does not necessarily have to be adopted.

[0091] As an example of an apparatus independently implemented as described herein, an image processing system including an eye detection unit, or an image processing method or program including an eye detection step may be provided. In this image processing system, the eye detection unit (or the eye detection step) sets an eye detection score in the eye detection region, and sets a detection score in the inner region of the detection frame or in the detection region based on the eye detection score. The setting of the detection score is performed by setting the eye detection region C k , C k-n , C k+m This can be implemented depending on the positional relationship between the detection frame and the detection area, and at least the first example (interpolation in FIG. 13, etc.) and the second example (region division in FIG. 17) can be adopted. The technology described here can also adopt the various modified examples described in the first and second embodiments. The adoptable modified examples include at least variations in the image detection process (replacement of the detection frame and detection area), variations in the shape of the detection frame or detection area, variations in the threshold value, variations in calculation such as interpolation, the number of region divisions, and processing based on rule (c) or (d).

[0092] While various embodiments of the present invention have been described, they are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as set forth in the claims. [Explanation of symbols]

[0093] 1: Image processing system, 2: Communication line, 3: User terminal, 11: Control unit, 11a: Image data acquisition unit, 11b, 111b: Image detection unit, 11c: Mask area calculation unit, 11d: Mask image generation unit, 12: Storage unit, 13: Communication unit, 14: Operation input unit, 15: Monitor, 111b1: Face image detection unit, 111b2: Eye detection unit, A1, A6, A7: Area (non-overlapping area), A2, A3, A4, A5: Area (overlapping area), A ave :Average score, A th1 : First threshold, B k , B k-n , B k+m , B k-n+1 , B k+m-1 : detection frame, B' k , B' k-n , B' k+m : Detection frame (face detection frame), CL: Virtual central axis, C k , C k-n , C k+m : Eye detection area, 12a: Image DB, 12b: Detection information DB, 12c: Mask information DB, 12d: Masked image DB, D k :Figure for calculating mask area, HL:Horizontal axis, M,:Mask, M k , M k+m : Mask area, P1: Detection target (face), P1a: Detection target (eye), P2: Non-detection target, R: Video, RM: Masked video, f k , f k-n , f k+m , f k+2m : Frame,

Claims

1. An image processing system including a mask area calculation unit, the mask area calculation unit acquires a detection result of an image detection process for an image of a frame in a moving image; the mask area calculation unit calculates a mask area for the image of the first frame by using at least a detection score value included in the detection result of the image of the second frame; the image detection process associates a detection frame or a detection region with at least the image of the second frame, and sets the detection score to the detection frame or the detection region; the detection result includes the detection frame or the detection area and the detection score; The second frame is a frame at a different point in time on the timeline of the video than the first frame.

2. 10. The system of claim 1, by the image detection processing, a first detection frame or a first detection area is associated with the image of the first frame, and a first detection score is set for the first detection frame or the first detection area; a second detection frame or a second detection area is associated with the image of the second frame, and a second detection score is set for the second detection frame or the second detection area; the mask region calculation unit calculates the mask region of the detection target in the first frame so that the mask region is larger in an object moving state than in an object stationary state when the first and second detection scores are equal to or greater than a first threshold; the object stationary state is such that the position of the detection target is the same between the first and second frames, The object movement situation is such that the position of the detection target is different between the first and second frames.

3. 3. The system according to claim 1 or claim 2, by the image detection processing, a first detection frame or a first detection area is associated with the image of the first frame, and a first detection score is set for the first detection frame or the first detection area; a second detection frame or a second detection area is associated with the image of the second frame, and a second detection score is set for the second detection frame or the second detection area; the mask region calculation unit calculates the mask region so as not to set a mask on at least a part of a non-overlapping region when the first and second detection scores are less than a first threshold; The non-overlapping region is the following region (a) or region (b): (a) a region inside the first or second detection frame where the first detection frame and the second detection frame do not overlap; (b) a region inside the first or second detection region where the first detection region and the second detection region do not overlap;

4. 3. The system according to claim 1 or claim 2, by the image detection processing, a first detection frame or a first detection area is associated with the image of the first frame, and a first detection score is set for the first detection frame or the first detection area; a second detection frame or a second detection area is associated with the image of the second frame, and a second detection score is set for the second detection frame or the second detection area; the mask area calculation unit calculates the mask area in the image of the first frame based on a figure for calculation; The system, wherein the graphic is determined based on the first and second detection frames or the first and second detection regions and the first and second detection scores.

5. 3. The system according to claim 1 or claim 2, By the image detection process, an eye detection area for detecting eyes included in the image is associated with the image of each frame; A system in which the image detection process sets the detection frame to surround the eye detection region, or sets the detection region to overlap the eye detection region.

6. 6. The system of claim 5, further comprising an eye detection unit for executing the image detection process; The eye detection unit setting an eye detection score for the eye detection region; The system sets the detection score for the inner region of the detection frame or the detection score for the detection region based on the eye detection score.

7. 7. The system of claim 6, The system, wherein the eye detection unit sets the detection score of the inner area of ​​the detection frame or the detection score of the detection area in accordance with the following rule (c) or (d): (c) When both eyes are detected for a face included in an image of a frame and the eye detection score is set to a second threshold or higher in only one of the eye detection areas of the both eyes, the eye detection score of the one of the eye detection areas is used to set the detection score of the inner area of ​​the detection frame or the detection score of the detection area, the detection frame is set to surround the one of the eye detection areas but not the other of the eye detection areas, and the detection area is set to overlap the one of the eye detection areas but not the other of the eye detection areas. (d) When only one eye is detected for a face included in an image of a frame, the detection score for the internal area of ​​the detection frame or the detection score for the detection area is set based on the eye detection score set in the eye detection area of ​​the one eye.

8. 6. The system of claim 5, the image detection process associates the eye detection region and the detection score with each of the eyes of a face included in each of the first and second frame images; the mask area calculation unit calculates the mask area so that a larger mask is provided on the side of the eye detection area for the left eye when the eye detection score for the left eye is higher than that of the right eye in each of the first and second frame images; The mask area calculation unit calculates the mask area so that, when the eye detection score is higher for the right eye than for the left eye in each of the first and second frame images, a larger mask is placed on the side of the eye detection area for the right eye.

9. 3. The system according to claim 1 or claim 2, the mask region calculation unit acquires a plurality of the second frames; The plurality of second frames include a frame that is later than the first frame and a frame that is earlier than the first frame.

10. An image processing method for causing a computer to execute a mask region calculation step, The mask region calculation step acquires a detection result of an image detection process for an image of a frame in a moving image, the mask area calculation step calculates a mask area for the image of the first frame using a detection score value included in the detection result of at least the image of the second frame; the image detection process includes associating a detection frame or a detection area with at least the image of the second frame, and setting the detection score for the detection frame or the detection area; the detection result includes the detection frame or the detection area and the detection score; The method, wherein the second frame is a frame at a different point in time on the timeline of the video than the first frame.

11. A program for causing a computer to execute a mask area calculation process, The mask area calculation process causes a computer to obtain a detection result of an image detection process for an image of a frame in a moving image; the mask area calculation process causes a computer to calculate a mask area for an image of a first frame using a detection score value included in the detection result of at least an image of a second frame; the image detection process includes associating a detection frame or a detection area with at least the image of the second frame, and setting the detection score for the detection frame or the detection area; the detection result includes the detection frame or the detection area and the detection score; The second frame is a frame at a different point in time on the timeline of the video from the first frame.

Images