Image processing system, image processing method, and program

The image processing system effectively removes subjects and corresponding audio components from videos using machine learning, addressing the quality deterioration issue in existing technologies by integrating subject and audio detection for seamless video editing.

WO2024241767A1PCT designated stage expired Publication Date: 2024-11-28CANON KK
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/015151
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-04-16
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Current video editing technologies deteriorate video quality when deleting a subject with audio, as residual audio components corresponding to the removed subject can still be played back, causing discomfort and quality issues.

Method used

An image processing system that detects and deletes specific subjects from video frames while simultaneously identifying and removing corresponding audio components from the audio data, using machine learning models for subject and audio type determination and vector calculation to ensure seamless deletion without quality loss.

Benefits of technology

Enables the removal of specific subjects from videos with audio while maintaining the quality of the video and audio, preventing the playback of residual audio components, thus enhancing user experience by ensuring a smooth and high-quality editing process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024015151_28112024_PF_FP_ABST
    Figure JP2024015151_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This image processing system is characterized by comprising: a subject detection means for detecting, among a plurality of frames of video data accompanied by voice data, a subject of a first type; a subject deletion means for deleting the subject of the first type from one or more frames, among the plurality of frames, in which the subject of the first type was detected; a voice detection means for detecting, in the voice data, a voice component corresponding to the subject of the first type; and a voice deletion means for deleting, from the voice data, the voice component corresponding to the subject of the first type.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing system, image processing method, and program

[0001] The present invention relates to an image processing system, an image processing method, and a program.

[0002] Digital cameras and smartphones with the ability to record video with audio are now widespread. However, videos with audio recorded by users may contain unwanted subjects. For example, when a user wants to record a person, a car may appear in the video, which is undesirable for the user.

[0003] Currently, there is known a technique for erasing unnecessary areas in an image so as to leave no trace of them (Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2007-286734

[0005] Consider the case where a subject making a sound is deleted from a video with audio. In this case, when the video with audio is played back, the audio still contains the audio components corresponding to the deleted subject, which is no longer displayed, which may cause the user to feel uncomfortable. In this way, deleting a subject making a sound from a video with audio reduces the quality of the video with audio.

[0006] The present invention has been made in consideration of the above circumstances, and provides a technology that makes it possible to delete a specific subject from a video with audio while suppressing degradation in the quality of the video with audio.

[0007] In order to solve the above problem, the present invention provides an image processing system comprising: a subject detection means for detecting a first type of subject in multiple frames of video data accompanied by audio data; a subject removal means for removing the first type of subject from one or more frames among the multiple frames in which the first type of subject is detected; an audio detection means for detecting audio components corresponding to the first type of subject in the audio data; and an audio removal means for removing the audio components corresponding to the first type of subject from the audio data.

[0008] According to the present invention, it is possible to remove a specific subject from a video with audio while suppressing degradation in the quality of the video with audio.

[0009] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiment of the present invention.

[0010] The accompanying drawings are incorporated in and constitute a part of the specification, illustrate embodiments of the present invention, and are used, together with the description, to explain the principles of the present invention. A diagram showing the hardware configuration of an image processing system. A diagram showing the functional configuration of an image processing system according to a first embodiment. A flowchart of image processing performed by the image processing system according to the first embodiment. A diagram explaining an example of object removal according to the first embodiment. A diagram explaining an example of object removal according to the first embodiment. A diagram explaining an example of object removal according to the first embodiment. A diagram explaining an example of audio component removal according to the first embodiment. A diagram showing the functional configuration of an image processing system according to a second embodiment. A flowchart of image processing performed by the image processing system according to the second embodiment. A diagram explaining an example of audio component separation according to the second embodiment.

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] [First embodiment] Hardware configuration of an image processing system Fig. 1A is a diagram showing the hardware configuration of an image processing system. In Fig. 1A, an information processing device 100 is a device equipped with a video editing function, such as a personal computer (PC) or a smartphone. The information processing device 100 includes a CPU 101, a ROM 102, a RAM 103, a HDD 104, a GPU 105, a network communication unit 106, an operation input unit 108, a display unit 109, an audio output unit 110, and a data communication unit 111. These components of the information processing device 100 are connected to each other via a system bus 107.

[0013] The CPU 101 uses the RAM 103 as a work area and executes programs stored in the ROM 102 or the HDD 104 to comprehensively control the operation of the information processing device 100. The programs executed by the CPU 101 include a video editing application program. The ROM 102 is a read-only nonvolatile storage medium that stores programs such as firmware. The RAM 103 is a volatile storage medium that can read and write information at high speed and is used as a work area when the CPU 101 processes information. The HDD 104 is a nonvolatile storage medium that can read and write information and stores an OS, various control programs, application programs, video data and audio data used in video editing, etc.

[0014] The GPU 105 cooperates with the CPU 101 to execute processes for video editing, learning and inference using machine learning technology, and the like. Generally, a GPU can perform more efficient calculations than a CPU by processing more data in parallel. Therefore, when the GPU 105 is used in addition to the CPU 101, inference regarding video and audio can be efficiently performed multiple times using a trained model in deep learning. Note that the inference process using the trained model, which will be described later, may be performed by either the CPU 101 or the GPU 105.

[0015] The network communication unit 106 is an interface for connecting to the server 130 via the network 120. The operation input unit 108 accepts operations from the user via a keyboard, mouse, touch panel, etc. The user can operate the video editing application through these operations. The display unit 109 is a monitor or display that displays a graphical user interface (GUI) of the information processing device 100. The display unit 109 also displays the GUI of the video editing application, and the user can edit videos by operating the GUI. The audio output unit 110 is an audio playback device such as a speaker. Alternatively, the audio output unit 110 may be an output terminal that can be connected to an audio playback device such as earphones or headphones. The user can hear the audio played back by the video editing application through the audio output unit 110.

[0016] The data communication unit 111 is an interface such as USB, SD, PCI Express, or SATA, and is capable of data communication with various recording media such as USB memory, SD cards, and SSDs. A user can import video data and audio data obtained by video shooting via the data communication unit 111 and store the data in the HDD 104 or the like. The user can then edit the video data and audio data stored in the HDD 104 or the like using a video editing application. Alternatively, the user can import video data and audio data from devices such as a camera, PC, or smartphone (not shown) via the network 120. The method for importing video data and audio data into the information processing device 100 is not particularly limited.

[0017] The server 130 is a server that shares part of the processing of the information processing device 100, and is, for example, a device such as a personal computer (PC). In this embodiment, the processing that the server 130 shares is not particularly limited, but may be, for example, processing related to video editing and machine learning.

[0018] The server 130 includes a CPU 131, a ROM 132, a RAM 133, a HDD 134, a GPU 135, and a network communication unit 136. These components of the server 130 are interconnected via a system bus 137. The functions of the CPU 131, the ROM 132, the RAM 133, the HDD 134, the GPU 135, and the network communication unit 136 are similar to those of the CPU 101, the ROM 102, the RAM 103, the HDD 104, the GPU 105, and the network communication unit 106 of the information processing device 100, respectively. However, the server 130 generally has more sophisticated and larger-capacity hardware resources than the information processing device 100. Therefore, when the hardware resources of the information processing device 100 alone are insufficient, the hardware resources of the server 130 can be used to perform processing more efficiently. However, all processing may be completed by the information processing device 100 alone. Therefore, although the image processing system illustrated in FIG. 1A includes the information processing device 100 and the server 130, the image processing system of this embodiment does not necessarily include the server 130.

[0019] Functional Configuration of Image Processing System Fig. 1B is a diagram showing the functional configuration realized by the hardware of the image processing system shown in Fig. 1A working in cooperation with a program (software). In Fig. 1B, the image processing system includes an area selection unit 141, an object type determination unit 142, an object vector acquisition unit 143, an object deletion unit 144, an audio type determination unit 145, an audio vector acquisition unit 146, a type match determination unit 147, and an audio deletion unit 148. The software of this embodiment also includes a video editing application. The video editing application operates when the CPU 101 executes a program stored in the ROM 102 or HDD 104 using the RAM 103 as a work area.

[0020] The region selection unit 141 selects an arbitrary region within the angle of view of an arbitrary frame (region selection frame) of the moving image data displayed on the display unit 109. For example, the region selection unit 141 selects an area designated by the user in accordance with an instruction from the user via the operation input unit 108. The region selection frame is, for example, a frame designated by the user in the moving image data.

[0021] The subject type determination unit 142 determines the type of subject (e.g., person, dog, automobile, or other) included in the area (selected area) selected by the area selection unit 141, and outputs information indicating the determined type. The determination of the subject type is realized, for example, by using an image of the selected area as input and performing inference using a trained model (first machine learning model) configured to identify the type of subject included in the input image.

[0022] In this embodiment, any known technology can be used for machine learning. For example, the object type determination unit 142 uses an image trained model. In generating the image trained model, an image to be identified is used as input data, and information on the object type of the image in the input data (e.g., person, dog, car, or other) is used as training data to generate an image trained model that outputs the type of object corresponding to the image. Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. Another example of an algorithm is deep learning, which uses a neural network to generate features and connection weighting coefficients for learning. Any of these algorithms can be used as appropriate and applied to this embodiment.

[0023] In the inference phase, the trained model for images takes an image of the selected area as input data and outputs information indicating the type of subject contained in the image (e.g., person, dog, car, or other).

[0024] In this embodiment, the hardware used to generate the trained model and perform inference based on the trained model is not particularly limited, and for example, the CPU 101, the GPU 135, the CPU 131, and some or all of the GPU 135 may be used. Also, a different device (not shown) may be used.

[0025] The subject vector acquisition unit 143 calculates the velocity vector of the subject detected through the type determination by the subject type determination unit 142. For example, the subject vector acquisition unit 143 tracks the subject in several frames before and after the region selection frame and calculates the velocity vector of the subject from the amount of movement of the subject. The subject can be tracked, for example, by detecting the subject in each frame using machine learning technology, as with the subject type determination unit 142. Alternatively, the subject may be tracked by pattern matching of pixel values ​​between frames without using machine learning technology. The hardware used to calculate the subject's velocity vector is not particularly limited, and for example, the CPU 101, the GPU 135, the CPU 131, and some or all of the GPU 135 may be used.

[0026] The object removal unit 144 removes the object detected by the object type determination unit 142 from the area selection frame. Simply removing the object from the area selection frame would result in an unnatural video. Therefore, the object removal unit 144 complements the background of the area from which the object has been removed by blending the object into the background. The object removal unit 144 also removes the object and complements the background in a similar manner in frames other than the area selection frame if a corresponding object is present within the angle of view. The hardware used to remove the object is not particularly limited, and may include, for example, the CPU 101, the GPU 135, the CPU 131, and some or all of the GPU 135.

[0027] The sound type determination unit 145 analyzes the sound data corresponding to the frame from which the object has been deleted by the object deletion unit 144, and outputs information indicating the type of sound included in the sound data (for example, person, dog, car, or other). The determination of the sound type is realized, for example, by using the sound data as input and performing inference using a trained model (second machine learning model) configured to identify the type of each object corresponding to each sound component included in the input sound data.

[0028] In this embodiment, any known technology can be used for machine learning. For example, the sound type determination unit 145 uses a trained model for sound. In generating the trained model for sound, the sound to be identified is used as input data, and information on the type of object corresponding to the sound of the input data (e.g., person, dog, car, or other) is used as training data to generate a trained model for sound that outputs the type of object corresponding to the sound. As with the object type determination unit 142, various algorithms can be used as specific algorithms for machine learning.

[0029] In the inference phase, the trained model for voice takes voice as input data and outputs information indicating the type of subject corresponding to each voice component contained in the voice (e.g., person, dog, car, or other).

[0030] In this embodiment, the hardware used to generate the trained model and perform inference based on the trained model is not particularly limited, and for example, the CPU 101, the GPU 135, the CPU 131, and some or all of the GPU 135 may be used. Also, a different device (not shown) may be used.

[0031] The sound vector acquisition unit 146 calculates the position and velocity vector of the sound. An example of a method for calculating the position and velocity vector of the sound is described below. For example, when sound data is recorded using two microphones, the sound vector acquisition unit 146 identifies the position of the subject's sound based on the difference in arrival time of the sound reaching the two microphones. The sound vector acquisition unit 146 then calculates the velocity vector of the subject's sound based on the movement of the position of the sound of the subject and the time axis of the sound data. Furthermore, a microphone array using three or more microphones or a directional microphone may be used to record sound data, making it easier to calculate the position and velocity vector of the sound source. The hardware used to calculate the velocity vector of the sound is not particularly limited, and may include, for example, the CPU 101, the GPU 135, the CPU 131, and part or all of the GPU 135. Alternatively, a different device (not shown) may be used.

[0032] The type matching determination unit 147 compares the type of object determined by the object type determination unit 142 with the type of sound determined by the sound type determination unit 145 to determine whether there is a sound component of a type matching the type of object to be deleted (e.g., person, dog, automobile, etc.). The type matching determination unit 147 also compares the position and velocity vector of the object calculated by the object vector acquisition unit 143 with the position and velocity vector of the sound calculated by the sound vector acquisition unit 146 to determine whether there is a sound velocity vector corresponding to the velocity vector of the object to be deleted. If there is a corresponding sound velocity vector, the sound vector acquisition unit 146 can determine that the sound (sound component) is of the same type as the object to be deleted. This is because, when the type of object cannot be correctly determined due to insufficient learning of the image learning model or the sound learning model described above, a different function, velocity vector calculation, can be used to detect the sound component corresponding to the object to be deleted. Furthermore, the type matching determination unit 147 may not use the velocity vector, but may use only the type information obtained by the object type determination unit 142 and the sound type determination unit 145. Alternatively, the type information obtained by the object type determination unit 142 and the sound type determination unit 145 may be used only the velocity vector. In this way, the method for identifying sound components corresponding to objects to be deleted from video data is not particularly limited, and various methods including the method described here may be used.

[0033] The audio deletion unit 148 separates and deletes audio components determined by the type matching determination unit 147 to correspond to the subject to be deleted from other audio components. Audio components other than the audio components corresponding to the subject to be deleted are not deleted. Any known technology can be used for separating and deleting audio components. To explain one example of multiple technologies, the audio deletion unit 148 determines the type of audio using a trained model for audio similar to that described with respect to the audio type determination unit 145, and separates the audio components for each type. At this time, the audio deletion unit 148 performs a Fourier transform on the audio data, treats the audio data as spectral information, masks the spectrum of the audio type to be deleted, and performs an inverse Fourier transform to restore the audio data, thereby being able to generate audio data from which only a specific audio type has been deleted.

[0034] Image Processing Flow Figure 2 is a flowchart of image processing executed by the image processing system. The target of image processing is video data accompanied by audio data. As mentioned above, the audio data and video data are recorded, for example, on the HDD 104. When the user of the information processing device 100 selects a function to delete a subject on the user interface of the video editing application, the processing of this flowchart begins.

[0035] The overall control of this flowchart is performed by the CPU 101. The processing of each step of this flowchart is performed by each unit shown in Fig. 1B. The hardware that realizes the functions of each unit shown in Fig. 1B is not particularly limited, and may be realized by, for example, the CPU 101, the GPU 135, the CPU 131, and some or all of the GPU 135, as long as it is technically possible.

[0036] In S201, the region selection unit 141 selects a specific region (selection region) in a specific frame (region selection frame) among multiple frames of video data. The selection region is, for example, a region designated by the user.

[0037] In S202, the subject type determination unit 142 determines the type (first type) of the subject (target subject) included in the selected area. This allows the target subject to be detected and its type to be identified. Additionally, the subject vector acquisition unit 143 may calculate (acquire) a velocity vector of the target subject across multiple frames.

[0038] In S203, the subject type determination unit 142 detects target subjects in other frames of the video data (frames other than the region selection frame).

[0039] In S204, the subject removal unit 144 removes the target subject from each of one or more frames (target frames) in which the target subject was detected by the subject detection in S201 or S202. Following the removal of the target subject, the subject removal unit 144 complements the area of ​​the removed subject by blending it into the background. For example, if the target subject is captured within the angle of view of the 201st to 300th frames in video data with 500 frames, the target subject is removed and the background is complemented for the 201st to 300th frames.

[0040] In S205, the sound type determination unit 145 determines the type of sound included in the sound data (the type of each object corresponding to each sound component). In addition, the sound vector acquisition unit 146 may calculate (acquire) a velocity vector across multiple frames of the sound component corresponding to the target object.

[0041] In S206, the type matching determination unit 147 detects audio components corresponding to the target subject in the audio data and determines whether or not the audio components corresponding to the target subject are present. If the audio components corresponding to the target subject are present, the process proceeds to S207; if not, the process of this flowchart ends.

[0042] The type matching determination unit 147 detects audio (detects audio components corresponding to the target subject) based on the type of the target subject determined in S202 and the type of audio determined in S205. For example, consider a case where the type of the target subject is "automobile" and the types of audio are "automobile" and "person." In this case, audio components corresponding to automobiles are included in the audio data, and the audio components corresponding to automobiles are detected as audio components corresponding to the target subject. Alternatively, the type matching determination unit 147 may detect audio components corresponding to the target subject using the velocity vectors acquired in S202 and S205 instead of or in addition to the type determined in S202 and S205. When the velocity vectors are used, the type matching determination unit 147 can detect audio components corresponding to the target subject in the audio data (for each frame) by comparing the velocity vector of the target subject with the velocity vector of the audio components corresponding to the target subject.

[0043] In S207, the audio deletion unit 148 separates and deletes the audio components corresponding to the target subject from the audio data. Audio components other than the audio components corresponding to the target subject are not deleted.

[0044] Even if the target subject is not included in the field of view during video capture, if the target subject is making a sound near the microphone of the imaging device (camera), the audio component of the target subject may be recorded in the audio data. Therefore, for audio data corresponding to frames in which the target subject was not detected in S203, audio components corresponding to the target subject may be detected in S206. Therefore, in S207, the audio deletion unit 148 can delete audio components corresponding to the target subject from the audio data even in frames in which the target subject is not included, if the audio components are present. For example, consider a case in which, in 500 frames of video data, the target subject is captured within the field of view of frames 201 through 300, and the audio components corresponding to the target subject are present in the audio data corresponding to frames 101 through 400. In this case, when the audio type determination unit 145 performs a process of determining the audio type of the entire audio data in S205, audio components corresponding to the target subject are detected from the portions of the audio data corresponding to frames 101 through 400. Therefore, the audio deletion unit 148 can delete the audio components corresponding to the target subject from the 101st to 400th frames in which the audio components corresponding to the target subject exist.

[0045] Furthermore, when determining whether or not a sound component corresponding to a target subject is present in a frame in which the target subject is not present, the period during which the target subject was present within the field of view may be taken into consideration. For example, the sound type determination unit 145 may determine the type of sound for each predetermined period of time for sound data corresponding to a period before and after the period during which the target subject was present within the field of view. The predetermined period is set, for example, as a period of a predetermined length (e.g., a 10-frame period) before and after the period during which the subject was present within the field of view. The sound type determination unit 145 may repeatedly set predetermined periods in order from the period closest to the period during which the subject was present within the field of view to the period furthest from the period during which the target subject was present within the field of view until the sound component corresponding to the target subject no longer exists. Alternatively, the sound type determination unit 145 may perform a calculation to predict a frame in which the sound component corresponding to the target subject will disappear based on the velocity vector calculated by the object vector acquisition unit 143 or the sound vector acquisition unit 146 or the transition in the volume of the sound corresponding to the same type of object as the type of the target subject, and then delete the sound component up to the predicted frame.

[0046] Example of Deletion of Target Object and Corresponding Audio Components An example of deletion of target objects and corresponding audio components will be described with reference to FIGS. 3A to 3C and 4A to 4B.

[0047] 3A shows an example of three consecutive frames of video data. In these three frames, a car 301 is moving from right to left. All other objects are stationary.

[0048] 2, it is assumed that the user specifies area 310 while the frame in the middle row of FIG. 3A (the nth frame) is displayed on display unit 109. Area selection unit 141 selects area 310 in response to the user's specification of area 310. This process corresponds to S210 in FIG. 2.

[0049] 3A is rectangular, the shape of the region 310 and the method for specifying the region 310 are not particularly limited. For example, the region 310 may be circular. Alternatively, a configuration may be adopted in which the user specifies the region 310 by drawing a freehand circle around the desired region.

[0050] The subject type determination unit 142 determines that the type of subject included in area 310 is an automobile. Then, the subject type determination unit 142 detects automobiles, which are target subjects, in other frames of the video data. As a result, automobiles 301 are also detected in the upper and lower frames of Figure 3A. This processing corresponds to S202 to S203 in Figure 2.

[0051] Next, the object removal unit 144 removes the automobile 301 from each frame in which it was detected, as shown in Fig. 3B. Then, the object removal unit 144 complements the area of ​​the automobile 301 to be removed by blending it into the background, as shown in Fig. 3C. This process corresponds to S204 in Fig. 2.

[0052] Fig. 4A is a conceptual diagram of the audio data corresponding to the three frames shown in Fig. 3A. The audio type determination unit 145 determines the type of audio contained in the audio data of these three frames. Then, the type match determination unit 147 detects the audio component corresponding to the automobile 301 in Fig. 3A (automobile audio 401). This process corresponds to S205 to S206 in Fig. 2.

[0053] Next, the audio deletion unit 148 deletes the audio component corresponding to the automobile 301 (automobile audio 401). As a result, as shown in Fig. 4B, the audio data corresponding to the three frames does not include the audio component corresponding to the automobile 301, but includes the audio components of the dog and the person. This process corresponds to S207 in Fig. 2.

[0054] In addition, if frames other than the three frames shown in Figure 4A also contain audio components corresponding to the automobile 301, the audio deletion unit 148 similarly deletes the audio components corresponding to the automobile 301 from these frames as well.

[0055] Summary of the First Embodiment As described above, according to the first embodiment, when a specific subject (a first type of subject) is deleted from video data accompanied by audio data, the audio components corresponding to the deleted subject are deleted from the audio data. Therefore, when playing back a video with audio, it is possible to prevent audio containing audio components corresponding to a subject that has been deleted and is therefore not displayed from being played back. Therefore, according to this embodiment, it is possible to delete a specific subject from a video with audio while suppressing degradation in the quality of the video with audio.

[0056] Note that the specific image processing procedure described above with reference to FIG. 2 is merely one example of a processing procedure for preventing playback of audio that contains audio components corresponding to deleted objects that are not displayed. Any configuration for deleting a specific object from video data accompanied by audio data and deleting audio components corresponding to the deleted object from the audio data is within the scope of the technical concept of this embodiment. Therefore, to generalize the first embodiment, the image processing system detects a specific object (a first-type object) in multiple frames of video data accompanied by audio data and deletes the first-type object from one or more frames in which the first-type object was detected. The image processing system also detects audio components corresponding to the first-type object in audio data and deletes the audio components corresponding to the first-type object from the audio data.

[0057] [Second Embodiment] In the first embodiment, a configuration was described in which an object to be deleted from video data is first determined, and then audio components corresponding to that object are deleted from the audio data. In contrast, in the second embodiment, a configuration is described in which audio components to be deleted from audio data are first determined, and then the object corresponding to those audio components is deleted from the video data. Note that in the second embodiment, the basic configuration, including the hardware configuration ( FIG. 1A ), of the image processing system is the same as in the first embodiment. The following mainly describes the differences from the first embodiment.

[0058] Functional configuration of the image processing system Fig. 5 is a diagram showing the functional configuration realized by the hardware of the image processing system shown in Fig. 1A working in cooperation with a program (software). In Fig. 5, the image processing system includes an audio type determination unit 501, an audio selection unit 502, an audio deletion unit 503, an object type determination unit 504, a type match determination unit 505, and an object deletion unit 506.

[0059] The sound type determination unit 501 has substantially the same functions as the sound type determination unit 145. However, the sound type determination unit 501 determines the type of sound contained in the sound data for a period designated by the user out of the entire period of the sound data or for the entire period, and outputs information indicating the type (for example, person, dog, car, or other).

[0060] The function of the audio selection unit 502 will be described later with reference to Fig. 6. The function of the audio deletion unit 503 is similar to that of the audio deletion unit 148.

[0061] The subject type determination unit 504 has roughly the same functions as the subject type determination unit 142. However, while the subject type determination unit 142 determines the type of subject included in a specific region of a specific frame, the subject type determination unit 504 analyzes all frames in the period corresponding to the audio components deleted by the audio deletion unit 503. Furthermore, because the user does not specify an area, the subject type determination unit 504 analyzes all pixels in the frame and outputs information including the type of each subject included in the frame (for example, person, dog, car, or other).

[0062] The function of the type matching determination unit 505 is similar to that of the type matching determination unit 147. The function of the object deletion unit 506 is similar to that of the object deletion unit 144.

[0063] Image Processing Flow Figure 6 is a flowchart of image processing executed by the image processing system. The target of image processing is video data accompanied by audio data. As in the first embodiment, the audio data and video data are recorded, for example, on the HDD 104. When the user of the information processing device 100 selects a function to delete a subject on the user interface of the video editing application, the processing of this flowchart begins.

[0064] The overall control of this flowchart is performed by the CPU 101. The processing of each step of this flowchart is performed by each unit shown in Fig. 5. There are no particular limitations on the hardware that realizes the functions of each unit shown in Fig. 5, and as long as it is technically possible, the functions may be realized by, for example, the CPU 101, the GPU 135, the CPU 131, and some or all of the GPU 135.

[0065] In S601 , the voice type determination unit 501 determines the type of voice contained in the voice data, separates the voice components by type, and displays the type of each voice component on the display unit 109 .

[0066] An example of the processing in S601 will now be described with reference to Fig. 7. The upper part of Fig. 7 is a conceptual diagram of the audio data to be processed. "ALL" conceptually indicates audio data including all audio components, with the horizontal axis representing time and the vertical axis representing volume. The lower part of Fig. 7 is a conceptual diagram of each separated audio component. Audio components whose type cannot be determined are separated as "other" audio components. The following description will be given taking as an example a case where audio data is separated into person A, person B, car A, dog A, and other audio components.

[0067] In S602, the audio selection unit 502 selects a specific audio component corresponding to a specific type from the multiple audio components separated in S601. Here, the audio selection unit 502 may select an audio component specified by the user. The following description will be given taking as an example a case where the user specifies an audio component corresponding to car A. Also, in video data having 500 frames, it is assumed that the audio component corresponding to car A is included in the audio data corresponding to the 101st to 400th frames.

[0068] In S603, the audio deletion unit 503 deletes the audio component selected in S602 (target audio component) from the audio data. Audio components other than the selected audio component are not deleted. For example, audio components corresponding to car A included in the audio data corresponding to the 101st to 400th frames are deleted.

[0069] In S604, the object type determination unit 504 determines the type of object included in the video data. For example, the object type determination unit 504 determines the type of object based on the 101st to 400th frames corresponding to the deleted audio components. In this embodiment, unlike the first embodiment, the area selection unit 141 does not select an area of ​​the frame. Therefore, the object type determination unit 504 analyzes all pixels in each frame and outputs information indicating the type of each object included in the analyzed frame (for example, person, dog, car, etc.).

[0070] Note that the target of the process for determining the type of subject in S604 is not limited to frames corresponding to deleted audio components. For example, the subject type determination unit 504 may determine the type of subject for all frames of the video data.

[0071] In S605, the type matching determination unit 505 determines whether or not an object corresponding to the type of the target audio component is present in the video data based on the determination result in S604. For example, if the audio component of car A is the target audio component (the audio component deleted in S603), the type matching determination unit 505 determines whether or not an automobile is included in the determination result in S604. If an object corresponding to the type of the target audio component is present in the video data, the process proceeds to S606; if not, the process of this flowchart ends.

[0072] In S606, the object deletion unit 506 deletes the object corresponding to the type of target audio component from each frame of the video data (a frame in which an object corresponding to the type of target audio component was detected through the processes of S604 and S605). As the object is deleted, the object deletion unit 506 complements the area of ​​the deleted object by assimilating it into the background.

[0073] Summary of the Second Embodiment As described above, according to the second embodiment, audio components corresponding to a specific subject (a first type of subject) are selected in the audio data, and the selected audio components are deleted from the audio data. Furthermore, the subject corresponding to the deleted audio components is deleted from the video data corresponding to the audio data. Therefore, when playing back a video with audio, it is possible to prevent audio containing audio components corresponding to a subject that has been deleted and is therefore not displayed from being played back. Therefore, according to this embodiment, it is possible to delete a specific subject from a video with audio while suppressing degradation in the quality of the video with audio.

[0074] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0075] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0076] This application claims priority based on Japanese Patent Application No. 2023-084704, filed on May 23, 2023, the entire contents of which are incorporated herein by reference.

[0077] 141: area selection unit, 142: object type determination unit, 143: object vector acquisition unit, 144: object deletion unit, 145: sound type determination unit, 146: sound vector acquisition unit, 147: type match determination unit, 148: sound deletion unit

Claims

1. An image processing system comprising: a subject detection means for detecting a first type of subject in multiple frames of video data accompanied by audio data; a subject removal means for removing the first type of subject from one or more frames of the multiple frames in which the first type of subject is detected; an audio detection means for detecting audio components in the audio data that correspond to the first type of subject; and an audio removal means for removing the audio components from the audio data that correspond to the first type of subject.

2. The image processing system of claim 1, wherein the subject detection means identifies a type of subject contained in a first area based on an image of a first area of ​​a first frame of the plurality of frames, and the first type is the type of the subject contained in the first area.

3. The image processing system of claim 2, wherein the subject detection means identifies the type of the subject contained in the first region by performing inference using a first machine learning model with an image of the first region as input.

4. An image processing system according to claim 2 or 3, further comprising: an area selection means for selecting the first area in the first frame in accordance with an instruction from a user.

5. An image processing system as described in any one of claims 2 to 4, further comprising: an object vector acquisition means for acquiring a velocity vector across the multiple frames of the object included in the first area; and an audio vector acquisition means for acquiring a velocity vector across the multiple frames of the audio component corresponding to the object included in the first area, wherein the audio detection means detects the audio component corresponding to the first type of object in the audio data by comparing the velocity vector of the object included in the first area with the velocity vector of the audio component corresponding to the object included in the first area.

6. The image processing system according to any one of claims 1 to 4, characterized in that the voice detection means detects the voice components in the voice data corresponding to the first type of subject by performing inference using a second machine learning model using the voice data as input.

7. The image processing system described in claim 1, characterized in that the voice detection means detects a plurality of voice components in the voice data, each of which corresponds to a different type of subject, by performing inference using a second machine learning model with the voice data as input, the image processing system further comprising a voice selection means for selecting one of the plurality of voice components, and the first type is a type of subject corresponding to the selected voice component from among the plurality of voice components.

8. An image processing method executed by an image processing system, comprising: a subject detection step of detecting a first type of subject in multiple frames of video data accompanied by audio data; a subject removal step of removing the first type of subject from one or more frames among the multiple frames in which the first type of subject is detected; an audio detection step of detecting audio components in the audio data that correspond to the first type of subject; and an audio removal step of removing the audio components corresponding to the first type of subject from the audio data.

9. A program for causing a computer to function as each of the means of the image processing system according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Device and method for erasing unnecessary area image

    JP2007286734A

  • Image processing apparatus, image processing method, and program

    JP2010246026A

  • Video audio recorder and video audio reproducer

    JP2012138930A

  • Video presentation processing apparatus and program therefor

    JP2021027487A

  • Image processing apparatus, imaging apparatus, image processing method, computer program, and storage medium

    JP2021132286A