Image processing device, image processing method, and recording medium

Abstract

This image processing device comprises: an acquisition unit that acquires imaging target information related to an imaging target; a processing unit that processes a first image to detect at least one conversion target other than the imaging target identified by using the imaging target information; and an output processing unit that generates and outputs a second image after the detected conversion target is converted to another target using a conversion rule. The conversion rule varies depending on at least some of attributes of the conversion target.

Description

Image processing device, image processing method, and recording medium

[0001] The present invention relates to an image processing device, an image processing method, and a program.

[0002] When shooting video for broadcasting at a television station or for distribution on a video sharing site, if a person who is unaware of the video being filmed, such as a passerby, appears in the video, there is a possibility that privacy will be violated. Patent Document 1 describes a live streaming system that can detect the faces of people in the video and apply mosaic processing to the detected faces according to predetermined conditions.

[0003] Japanese Patent Application Laid-Open No. 2023-72567

[0004] However, there is a problem in that moving images in which people are pixelated, as in the system described in Patent Document 1, look strange to the viewer and are not very appealing.

[0005] The image processing device of the present disclosure comprises: an acquisition means for acquiring subject information relating to a subject to be photographed; a detection means for detecting at least one conversion subject other than the subject to be photographed identified using the subject information by processing a first image; and an output processing means for generating and outputting a second image after converting the detected conversion subject into another subject using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion subject.

[0006] The image processing method of the present disclosure includes one or more computers acquiring object information relating to an object to be photographed, processing a first image to detect at least one conversion object other than the object to be photographed identified using the object information, and generating and outputting a second image after converting the detected conversion object into another object using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion object.

[0007] The program in the present disclosure causes a computer to execute an acquisition process for acquiring subject information relating to a subject to be photographed; a detection process for detecting at least one conversion subject other than the subject to be photographed identified using the subject information by processing a first image; and an output process for generating and outputting a second image after converting the detected conversion subject into another subject using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion subject.

[0008] According to the present disclosure, an image processing device, an image processing method, and a program are provided that protect the privacy of objects that are not the subject of the image and that generate natural, attractive images.

[0009] FIG. 1 is a functional block diagram showing an example configuration of an image processing device according to the present disclosure; FIG. 2 is a flowchart showing an example processing operation of an image processing device according to the present disclosure; FIG. 3 is a diagram conceptually showing an example system configuration of an image processing system according to the present disclosure; FIG. 4 is a diagram showing an example configuration of a computer that realizes an image processing device according to the present disclosure; FIG. 5 is a diagram for explaining image conversion; FIG. 6 is a diagram showing an example of a conversion screen; FIG. 7 is a diagram showing an example of a setting screen; FIG. 8 is a diagram showing another example of the setting screen; FIG. 9 is a diagram conceptually showing an example system configuration of a second image processing system according to the present disclosure.

[0010] Hereinafter, in this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are denoted by similar reference numerals, and descriptions thereof will be omitted as appropriate. In each of the following drawings, configurations of parts that are not related to the essence of this disclosure are omitted and are not shown.

[0011] In the present disclosure, "acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition); and a device inputting data or information output from another device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a response, and accessing and reading information from another device or storage medium. Examples of passive acquisition include receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0012] <Example of Functional Configuration> As shown in Fig. 1, the image processing device 100 includes an acquisition unit 102, a detection unit 104, and an output processing unit 106. The acquisition unit 102 acquires shooting object information related to the shooting object. The detection unit 104 processes the first image to detect at least one conversion object other than the shooting object identified using the shooting object information. The output processing unit 106 generates and outputs a second image after converting the detected conversion object into another object using a conversion rule. The conversion rule differs depending on at least some of the attributes of the conversion object.

[0013] 2, in the image processing device 100, the acquisition unit 102 first acquires object information relating to the object (step S101). The detection unit 104 then processes the first image to detect at least one conversion object other than the object identified using the object information (step S103). The output processing unit 106 then converts the detected conversion object into another object using a conversion rule that differs depending on at least some of the attributes of the conversion object, thereby generating and outputting a second image (step S105).

[0014] According to this image processing device 100, the detection unit 104 processes the first image to detect at least one conversion target other than the subject to be photographed identified using the subject information, and the output processing unit 106 generates and outputs a second image after converting the detected conversion target into another target using a conversion rule that differs depending on at least some of the attributes of the conversion target. This protects the privacy of subjects other than the subject to be photographed that exist in the image, and makes it possible to generate a natural, attractive image.

[0015] A detailed example of the image processing device 100 will be described below.

[0016] (First embodiment) <System overview> Fig. 3 is a diagram conceptually illustrating the system configuration of an image processing system 1 according to the present disclosure. The image processing system 1 has a function for protecting the privacy of objects other than the subject of the video that appear in the video. The video may be, for example, video for broadcast by a television station or video for distribution on a video sharing site. The subject of the video may include, for example, performers in the video, people who have consented to the filming and broadcasting or distribution of the video, and filming locations and objects that have permission to film and broadcast or distribute the video. Objects other than the subject of the video may include, for example, people who appear in the video but are unaware that they are being filmed, people who have not consented to the filming and broadcasting or distribution of the video, and filming locations and objects that have not permission to film and broadcast or distribute the video.

[0017] The image processing system 1 includes an image processing device 100 and a storage device 120. The image processing device 100 is a computer such as a personal computer or a server computer. Alternatively, the image processing device 100 may be a computer that implements a communication device such as a smartphone, a tablet terminal, or a mobile phone. The image processing device 100 may also be implemented by combining multiple computers. The functions of the image processing device 100 may be implemented by installing an application program on various computers and launching the program. Alternatively, the functions of the image processing device 100 may be implemented in a form provided to a user by accessing a server computer on a cloud via a network such as the Internet from a communication device such as an operation terminal (for example, SaaS (Software as a Service)).

[0018] The storage device 120 may be provided inside or outside the image processing device 100. In other words, the storage device 120 may be hardware that is integrated with the image processing device 100, or may be hardware that is separate from the image processing device 100.

[0019] The storage device 120 includes a shooting target storage unit 130 , a moving image storage unit 140 , and a converted image storage unit 150 .

[0020] The photographic object storage unit 130 stores at least one of an image of the photographic object and feature information extracted from the image. As described above, the photographic object includes a person, a photographing location, and an object. The feature information stored in the photographic object storage unit 130 is information that can identify the photographic object. For example, if the photographic object is a person, the photographic object storage unit 130 stores a facial image of the person and feature information extracted from the facial image. If the photographic object is a location, the photographic object storage unit 130 stores images of buildings included in the location (especially images of signs attached to utility poles, etc., including house numbers, store names, company names, building names, etc., and house nameplates), feature information extracted from the image, or text information indicating characters included in the image. If the photographic object is an object, the photographic object storage unit 130 stores an image of the object, feature information extracted from the image, and, if the object contains characters that can identify the object, text information indicating the characters. These characters that can identify the object include, for example, characters indicating a vehicle license plate number or a product name.

[0021] Representative algorithms for extracting feature information include SIFT (Scale-Invariant Feature Transform) and HOG (Histograms of Oriented Gradients). These algorithms detect feature points from a given image and describe feature quantities calculated from pixel values ​​and differential values ​​of the points. However, the method for extracting feature information is not limited to this. As described above, the image processing device 100 may extract and acquire text information from an image using an OCR (Optical Character Reader).

[0022] When a photographing subject stored in this photographing subject storage unit 130 is detected from image data of the image processing target, it is excluded from image processing (conversion processing) by the image processing device 100. The image processing device 100 performs conversion processing on any subject that is detected from the image data of the image processing target and is not stored in this photographing subject storage unit 130.

[0023] The video storage unit 140 stores video data before image processing by the image processing device 100. The video data includes video for television broadcasting and video distributed over a network such as the Internet. However, the data subjected to image processing in this system is not limited to video and may also be still images.

[0024] The converted image storage unit 150 stores the video data after image processing by the image processing device 100. Here, the image processing includes converting objects other than the subject in the image into other objects so that the objects cannot be identified, thereby converting them into second image data. Furthermore, the image processing may include various analytical processes of the image data, such as performing pattern recognition based on predetermined conditions, determining whether the image is good or bad by comparing it with a reference value, recognizing the object by matching it with the feature amount of a predetermined object, and measuring the size of the object.

[0025] The image processing device 100 according to the present disclosure converts objects other than the subject of a photograph in an image into other objects so that the objects cannot be identified, and converts the converted objects into second image data. The conversion process into other objects includes, for example, converting a person into another fictional person, converting a vehicle license plate number into another fictional license plate, converting the scenery of the photographed location into another fictional scenery, etc. The type of object to be converted may be specified by an operator's operation. As will be described later, the body part, behavior, etc. of the object to be converted may also be specified by an operator's operation.

[0026] <Hardware Configuration Example> FIG. 4 is a block diagram illustrating a hardware configuration of a computer 1000 that realizes the image processing device 100 according to the present disclosure.

[0027] The computer 1000 has a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .

[0028] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0029] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0030] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0031] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the image processing device 100 (e.g., the acquisition unit 102, the detection unit 104, and the output processing unit 106). The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 may also function as the storage device 120 of the image processing device 100.

[0032] The program module may be recorded on a recording medium. The recording medium on which the program module is recorded may include a non-transitory, tangible medium usable by the computer 1000, and the program code readable by the computer 1000 (processor 1020) may be embedded in the medium.

[0033] The input / output interface 1050 is an interface for connecting the computer 1000 with various input / output devices. The input / output interface 1050 also functions as a communication interface for performing short-range wireless communication such as Bluetooth (registered trademark) and NFC (Near Field Communication).

[0034] The network interface 1060 is an interface for connecting the computer 1000 to a communication network. This communication network may be, for example, a local area network (LAN) or a wide area network (WAN). The network interface 1060 may connect to the communication network wirelessly or by wire.

[0035] The computer 1000 is then connected to necessary devices (for example, the display of the image processing device 100, a touch panel, operation buttons, a touchpad, a keyboard, a mouse, a speaker, a microphone, a camera, a printer, etc.) via the input / output interface 1050 or the network interface 1060.

[0036] Each component of the image processing device 100 according to the present disclosure in each figure is realized by any combination of hardware and software of the computer 1000 in Figure 4. Those skilled in the art will understand that there are various variations in the realization method and device. The functional block diagrams showing the image processing device 100 according to the present disclosure in each figure show blocks of logical functional units, rather than a configuration in hardware units.

[0037] The information processing device of the present invention may be realized as a single chip or device, or may be realized in a form that can be attached to another device. For example, the image processing device 100 may be built into a camera that generates an image to be converted.

[0038] <Example of functional configuration> The following will be described with reference to Fig. 1. As shown in Fig. 1, the image processing device 100 includes an acquisition unit 102, a detection unit 104, and an output processing unit 106. The acquisition unit 102 acquires shooting subject information related to the shooting subject. Specifically, the acquisition unit 102 acquires the shooting subject information from the shooting subject storage unit 130. As will be described later, a setting screen can be used to accept designation of the shooting subject, and the acquisition unit 102 reads and acquires information related to the designated shooting subject from the shooting subject storage unit 130.

[0039] For example, if the subject of photography is a person, the acquisition unit 102 acquires, from the photographic subject storage unit 130, a facial image of the person and feature information extracted from the facial image. If the subject of photography is a place, the acquisition unit 102 acquires, from the photographic subject storage unit 130, at least one of images of buildings included in the place (particularly, images of signs attached to utility poles or the like, including house numbers, store names, company names, building names, etc., and house nameplates), feature information extracted from the images, or text information indicating characters included in the images. If the subject of photography is an object, the acquisition unit 102 acquires, from the photographic subject storage unit 130, an image of the object, feature information extracted from the images, and, if the object contains characters that can identify the object, at least one of text information indicating the characters. These characters that can identify the object are, for example, characters indicating a vehicle license plate number, a product name, etc.

[0040] The shooting object storage unit 130 stores information (e.g., person's name, object's name, building name, store name, etc.) that can identify the shooting object, in association with at least one of an image and characteristic information. The setting screen accepts input of information that can identify the video shooting object. Upon accepting the input, the acquisition unit 102 searches for corresponding information from the shooting object storage unit 130. If an image is associated with the information, the acquisition unit 102 may display the image and have the operator confirm it, and then accept a designation of the object as the shooting object. The acquisition unit 102 reads information about the designated shooting object from the shooting object storage unit 130 and acquires it as shooting object information. Alternatively, the acquisition unit 102 may display a list of images of shooting objects registered in the shooting object storage unit 130 and accept an operator's selection operation for an object to be shot this time from the list. The acquisition unit 102 may then acquire an image or characteristic information of the selected object as shooting object information related to the shooting object. The list display may also be narrowed down by keyword or the like, or categorized by type.

[0041] As shown in Fig. 5, the first image 20 before image processing includes a person who is the shooting subject 30 and a person other than the shooting subject 30. The detection unit 104 processes the first image 20 to detect at least one conversion target 40a, 40b (hereinafter, when there is no particular need to distinguish between them, simply referred to as the conversion target 40) that is other than the shooting subject 30 identified using the shooting subject information. In Fig. 5, the detected conversion target 40 is indicated by a dashed line.

[0042] The conversion target 40 is a face of a person other than the photographing target 30. The detection unit 104 first detects a person's face from the first image 20. The detection unit 104 then extracts feature information from the detected face area and performs a process of comparing the feature information with feature information registered in the photographing target storage unit 130. The detection unit 104 identifies a face area that shows a degree of match with the feature information registered in the photographing target storage unit 130 that is equal to or greater than a threshold as the face of the person in the photographing target 30. On the other hand, the detection unit 104 determines a face area that shows a degree of match with the feature information registered in the photographing target storage unit 130 that is less than the threshold as the face of a person other than the photographing target 30, and identifies it as the conversion target 40.

[0043] The threshold for the result of matching with the feature information registered in the filming target storage unit 130 may be changed depending on the purpose of the video. For example, if the video is to be used in a news program, it is preferable to set a stricter threshold for the former than for the former. This is because video for a news program is likely to include people who are not aware that filming is taking place or who have not consented to appearing in the video, such as when the video is filmed at the scene of a sudden incident or accident. On the other hand, video for a variety show is likely to be filmed in a situation where people who may appear in the video are aware in advance that filming is taking place, or where people around them have consented to appearing in the video.

[0044] That is, in the case of footage for a news program, the threshold value is set higher so that all objects that do not strictly match the filming target 30 become the conversion target 40. Conversely, in the case of footage for a variety program, the threshold value may be set lower so that objects that match the filming target 30 to a certain extent are excluded from the conversion target 40.

[0045] The output processing unit 106 converts the detected conversion target 40 into another target (shown as modified images 50a and 50b in the figure) using the conversion rule, and generates and outputs the second image 22. As shown in FIG. 5 , in the second image 22, the modified image 50 is preferably superimposed with an image element that makes it possible to identify that it has been modified. Preferably, the superimposed display of this image element can be switched between display and non-display by an operator. In addition to displaying the modified image 50 in a frame, the superimposed display may also include displaying the area with a hatch, displaying a blinking frame, or displaying an image element such as an icon or mark indicating that the image has been modified in association with the modified image 50.

[0046] The conversion rule has parameters and image templates for conversion preset according to the attributes to be converted, for example. Details will be described later. Note that image processing (conversion processing) using the conversion rule may be executed by the image processing device 100 or may be executed by another external image processing device. In the following explanation, it is assumed that the image processing device 100 executes the conversion rule. Here, "conversion to another object" refers to converting the character when the object is a person, and does not mean converting another object, for example, a "person" into an "object."

[0047] The conversion rule includes attribute items to be converted. As shown in Fig. 7, the image processing device 100 can accept a specification (conversion setting) of whether or not to convert each attribute item by an operator's operation on a setting screen 400. The image processing device 100 can display the setting screen 400 on a display (not shown). The display can be, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like.

[0048] Details of the conversion rules when the conversion target 40 is a person will be described later. Here, a case where the conversion target 40 is a vehicle license plate, characters on a building sign, or the like will be described. First, the detection unit 104 detects characters on the vehicle license plate or the building sign from the first image 20 using an OCR (Optical Character Reader) to generate text data. Then, the detection unit 104 determines whether the vehicle license plate or the characters on the building sign indicated by the detected text data are registered in the photographed object storage unit 130. If they are not registered, the detection unit 104 identifies the target as the conversion target 40.

[0049] When converting the identified conversion target 40, the converted license plate number, building name, store name, etc. must be fictitious. Therefore, multiple candidates for the converted license plate number, building name, store name, etc. are pre-registered in the storage device 120 as templates. In the case of a vehicle license plate, the image processing device 100 converts at least the area name to a pre-registered fictitious place name. Furthermore, the image processing device 100 converts the number of digits in the vehicle license plate from four to five, for example, to a non-existent number. The combination of numbers may be set randomly or may be a pre-registered predetermined combination. Similarly, multiple fictitious name candidates for building names and store names are pre-registered in the storage device 120, and the image processing device 100 selects a name from the candidates and performs image processing to convert the character string of the conversion target 40 into the candidate name.

[0050] The following describes a case where the conversion target 40 is a landscape. Regarding the landscape conversion process, the image processing device 100 preferably accepts a setting in advance, via a settings menu (not shown), as to whether or not to perform the landscape conversion process. Furthermore, it is preferable that a plurality of replacement background image templates are pre-registered in the storage device 120. When the setting to perform the landscape conversion process is selected, the image processing device 100 converts the background image of the first image 20 using the templates. In this case, the image processing device 100 may accept, via a settings menu, a selection of which background image template to use. Alternatively, the settings menu may include a UI that accepts an update operation to change the automatically selected background image to another background image.

[0051] The setting screen 400 includes a shooting target setting field 410, a conversion setting field 420, and a setting save button 430. The shooting target setting field 410 accepts the specification of the above-mentioned shooting target 30 and displays the currently specified shooting target 30. In the example shown in the figure, only one shooting target 30 is specified, but multiple shooting targets 30 may be specified. For example, the shooting target setting field 410 of the setting screen 400 may further include a UI (User Interface) for adding registrations, and may accept the specification of additional shooting targets 30 by accepting an operation of the UI.

[0052] The conversion setting field 420 includes a UI (checkbox) for each attribute item that accepts the setting of the attribute item for which conversion processing is to be performed on the conversion target 40. The image processing device 100 performs conversion processing on the selected attribute item. Specifically, when the target is a person, the attributes of the conversion setting include at least one of gender, age, appearance, clothing, and behavior. When the attribute item gender is selected, the conversion rule includes, for example, converting male to female and female to male. When the attribute item age is selected, the conversion rule includes, for example, estimating the age of the conversion target 40 and converting the face, hairstyle, etc. of the conversion target 40 to an age obtained by changing the estimated age by a predetermined age (e.g., adding or subtracting 10 years).

[0053] When the attribute item "appearance" is selected, the conversion rule includes, for example, conversion of the face, hairstyle, etc. of the conversion target 40. The conversion process for appearance includes, for example, conversion of a face to the face of a fictitious different person, conversion of a hairstyle from long to short, conversion of hair color from black to brown, and conversion of straight hair to perm. When the attribute item "clothing" is selected, the conversion rule includes, for example, conversion of the clothing of the conversion target 40. The conversion process for clothing includes conversion of clothing to a fictitious different clothing, conversion of short sleeves to long sleeves, conversion of a skirt to pants, and conversion of accessories. When the attribute item "behavior" is selected, the conversion rule includes, for example, conversion of the behavior of the conversion target 40. The conversion process for behavior includes, for example, conversion of behavior to a fictitious different behavior, hiding (deleting from the first image 20) a person who has been detected performing a predetermined behavior, and predetermined correction processing.

[0054] When the setting save button 430 is pressed, the image processing device 100 stores the content set on the setting screen 400 in the storage device 120 .

[0055] The image processing device 100 may also accept more detailed settings using an attribute setting screen 500 shown in FIG. 8 . The attribute setting screen 500 includes an attribute setting field 510 in addition to the setting screen 400 shown in FIG. 7 . The attribute setting field 510 includes a UI (check box) for specifying whether or not to perform conversion processing for each predetermined action or behavior when the predetermined action or behavior is detected. The predetermined action or behavior includes illegal actions, immoral actions, actions that cause viewers discomfort, and actions that interfere with viewing of the video. The conversion rule includes performing processing such as hiding the predetermined action or behavior or converting it into another image when the predetermined action or behavior is detected. The predetermined action and behavior shown in FIG. 8 are merely examples and are not limiting.

[0056] Specifically, the conversion rules include a process of hiding the conversion target 40 when a behavior of getting too close to the camera is detected (for example, when the distance between the eyes of the conversion target 40 is below a threshold). The conversion rules include a process of hiding the conversion target 40 when a behavior of invading a predetermined area is detected or a behavior of getting too close to each other (the conversion target 40 is multiple people). Furthermore, the conversion rules include a process of correcting the conversion target 40 by putting clothes on it when a naked conversion target 40 is detected. The conversion rules may also include a process of erasing the tattoos of the conversion target 40 or a process of hiding the person (deleting the person from the first image 20) when a tattooed person is detected.

[0057] As described above, templates for identifying predetermined actions that particularly require conversion processing are registered in advance in the storage device 120. The image processing device 100 uses the templates to detect the predetermined actions from the conversion target 40 in the first image 20 and performs the conversion processing.

[0058] For each of the attribute items of appearance, clothing, and behavior, at least one post-conversion template image may be registered in advance. The image processing device 100 may select one of the template images to use as the modified image 50 of the conversion target 40. For example, as shown in FIG. 6 (to be described later), the conversion screen 300 may be configured to reselect this template image each time the update button 332 is pressed.

[0059] The image processing device 100 of the present disclosure is intended for video. Therefore, for example, after a conversion target 40 is identified in the first image 20, the image processing device 100 may acquire feature information of the conversion target 40 and store it in the storage device 120 in order to track the conversion target 40. For example, the image processing device 100 extracts feature information for each of the face region and body region of the conversion target 40 and stores it in the storage device 120. The image processing device 100 then tracks the conversion target 40 using the feature information of the conversion target 40 (using the feature information of the face region if a face is detected, and using the feature information of the body region if a face is not detected). In this way, the conversion target 40 is tracked, a conversion process is performed on the conversion target 40, and the corrected image 50 can be made to follow it. In other words, the corrected image 50 corresponding to each conversion target 40 can be kept constant while being followed. This makes the converted second image 22 more natural to view.

[0060] <Operation Example> The following will be described with reference to Fig. 2, Fig. 5 to Fig. 7, etc. First, the image processing device 100 displays a setting screen 400 on the display, as shown in Fig. 7, and accepts settings for the subject 30 to be photographed and the conversion settings in response to pressing a setting save button 430. However, the setting screen 400 may also include a cancel button for not updating the settings, a close button for closing the setting screen 400, or the like.

[0061] The settings on this setting screen 400 may be applied as default settings to all conversion targets 40. In another example, the setting screen 400 may allow attribute items to be set for each group set using a predetermined criterion. In this example, the conversion process is performed using conversion rules set in accordance with the settings on the setting screen 400.

[0062] The predetermined criteria may be, for example, a person under a predetermined age (for example, a minor), a foreigner, or the like.

[0063] This process can be started at various times, such as when the operator selects the start operation in the operation menu, when it is detected that the start shooting operation has been performed on the camera, when an image generated by the camera is acquired, when the image is saved in the video storage unit 140, or when a specified time has arrived, but is not limited to these.

[0064] First, the acquisition unit 102 acquires the photographing object information about the photographing object 30 (step S101). The acquisition unit 102 acquires the photographing object information (characteristic information, etc.) about the photographing object 30 specified on the setting screen 400 described above.

[0065] Then, the detection unit 104 processes the first image 20 to detect at least one conversion target 40 other than the shooting target 30 identified using the shooting target information (step S103). As shown in Fig. 5, the detection unit 104 detects a person from the first image 20. The method for detecting a person is not particularly limited, but possible methods include detecting facial features (eyes, nose, mouth) and recognizing that the person is a human face based on the positions of the features.

[0066] The detection unit 104 then extracts feature information from the detected person and compares the extracted feature information with the shooting target information (feature information of the shooting target 30). If the result of the comparison shows that the degree of match is equal to or greater than a threshold, the detection unit 104 identifies the person as the shooting target 30 and excludes the person from the conversion target. On the other hand, if the result of the comparison shows that the degree of match is less than the threshold, the detection unit 104 identifies the person as a person other than the shooting target 30 and as the conversion target 40.

[0067] Then, the output processing unit 106 generates and outputs the second image 22 after converting the detected conversion object 40 into another object using the conversion rule (step S105).

[0068] Here, as shown in Figure 6, before generating the second image 22, the output processing unit 106 displays an image of the face of the person that is the conversion target 40 and a modified image 50 of the modified face of the person after converting the conversion target 40 side by side on the same screen (conversion screen 300).

[0069] The conversion screen 300 includes a before-conversion display field 310, a conversion setting display field 320, and a after-conversion display field 330. The before-conversion display field 310 displays an image of the conversion target 40 detected in the first image 20 before the conversion process. However, the before-conversion display field 310 may also display all objects (detected faces) other than the conversion target 40. In this way, it is possible to set conversion settings and perform the conversion process for objects that the operator has determined to require conversion among the objects not identified as the conversion target 40 by the detection unit 104.

[0070] Here, the conversion rule includes at least one setting item. The conversion setting display field 320 of the conversion screen 300 further includes a conversion setting display field 320 (accepting means) that accepts setting input of at least one setting item for each attribute of the person who is the conversion target 40, in association with the conversion target 40. In this way, the conversion setting display field 320 displays the contents of the conversion process to be executed on the conversion target 40. The conversion setting contents initially set on the setting screen 400 shown in FIG. 7 are initially displayed. The conversion setting display field 320 may further include a UI that accepts an operation to change the conversion setting contents, and a check box that accepts selection of attribute items to be subject to the conversion process may be provided for each item.

[0071] When the output processing unit 106 receives a setting input through the conversion setting display field 320 (receiving means) associated with the conversion target 40, it updates the edited face (edited image 50) of the person who is the conversion target 40 using a conversion rule in accordance with the input setting items, and displays the updated face on the screen. The timing for updating the display of the edited image 50 may be when the input setting for the setting items is received, or when the update button 332 is pressed.

[0072] The post-conversion display field 330 displays the corrected image 50 after the conversion process. When the update button 332 is pressed, the output processing unit 106 may update the post-conversion display field 330 to display the corrected image 50 that has been subjected to the conversion process using a conversion rule in accordance with the settings displayed in the conversion setting display field 320. Furthermore, each time the update button 332 is pressed, the image processing device 100 may further change the parameters of each setting (for example, changing the change range for age) or the template (changing to a different appearance or clothing) and perform the conversion process.

[0073] The output processing unit 106 may display a correction notification field 342 associated with the updated corrected image 50 so that the updated corrected image 50 can be identified as having been updated. The correction notification field 342 may display a message indicating that the image has been corrected (e.g., "corrected"), but is not limited to this. For example, to indicate that the image has been updated, the display frame of the corrected image 50 may be highlighted by changing it to a different color from the other display frames, or an image element such as a label or icon that says "updated" may be associated with the corrected image 50 and superimposed on it.

[0074] In addition, the output processing unit 106 may use the above-mentioned predetermined criteria to group the faces of multiple people that are the conversion target 40, and use conversion rules set on a group-by-group basis to display on the screen images of the modified faces of the multiple people after converting the conversion target on a group-by-group basis.

[0075] For example, the output processing unit 106 identifies people (e.g., minors) who are under a predetermined age from among the conversion targets 40, and groups the people. Then, the output processing unit 106 displays the corrected images 50 (in the figure, corrected image 50a corresponding to conversion target 40a and corrected image 50b corresponding to conversion target 40b are shown, but when there is no particular need to distinguish between them, they are simply referred to as corrected images 50) that have been subjected to a conversion process set for each group (e.g., adding 10 years to the age or hiding the person) in the post-conversion display field 330 of the conversion screen 300 in Fig. 6 or the second image 22 in Fig. 5.

[0076] As described above, according to this embodiment, the image processing device 100 includes an acquisition unit 102, a detection unit 104, and an output processing unit 106. The acquisition unit 102 acquires shooting object information related to the shooting object. The detection unit 104 processes the first image to detect at least one conversion object other than the shooting object identified using the shooting object information. The output processing unit 106 generates and outputs a second image after converting the detected conversion object into another object using a conversion rule. The conversion rule differs depending on at least some of the attributes of the conversion object. This image processing device 100 can protect the privacy of objects other than the shooting object that are present in the image, while generating a natural, attractive image.

[0077] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0078] (Other Embodiments) <Processing for Reflecting Facial Expressions> In another embodiment, when performing a conversion process on a person who is the conversion target 40, the image processing device 100 may perform a process for reflecting the facial expression of the conversion target 40 before conversion in the facial expression of the person in the converted corrected image 50. The output processing unit 106 outputs the second image 22 in which information related to the facial expression of the person who is the conversion target 40 before conversion is reflected in the converted corrected face (corrected image 50) of the person who is the conversion target 40.

[0079] Specifically, the image processing device 100 identifies the facial expression of the conversion target 40 detected from the first image 20, and reflects the identified facial expression in the facial expression of the corresponding person in the corrected image 50. If the person before conversion has a crying face, the person after conversion will also have a crying face. The facial expressions may be, for example, but are not limited to, a smiling face, a crying face, an angry face, a surprised face, or any of a variety of other expressions.

[0080] Furthermore, when performing conversion processing on the person who is the conversion target 40, the image processing device 100 may perform processing to reflect the facial expression of the conversion target 40 before conversion into a different facial expression in the corrected image 50. In this case, the image processing device 100 can also generate the corrected image 50 in which the crying face of the conversion target 40 before conversion is converted into a smiling face.

[0081] According to this configuration, the facial expression of the conversion target 40 before conversion can be reflected in the converted modified face (modified image 50), so that the second image 22 can be generated in a way that is more natural to the viewer.

[0082] <Display Process of Multiple Candidates> In the above embodiment, a configuration has been described in which a post-conversion edited face (edited image 50) is displayed one-to-one with respect to the pre-conversion target 40, as shown in Fig. 6. In another embodiment, the output processing unit 106 may display, on the conversion screen 300, a plurality of edited faces that have been subjected to a conversion process using different parameters or different templates for the pre-conversion target 40.

[0083] The output processing unit 106 displays, on the same screen (conversion screen 300), a plurality of different modified faces (modified images 50) of the person who is the conversion target 40. Then, the output processing unit 106 accepts the selection of one of the plurality of modified faces on the conversion screen 300, and outputs a second image 22 in which the selected modified face is the modified face of the conversion target 40 after conversion.

[0084] As shown in the conversion screen 300 of Fig. 9, the conversion screen 300 includes a modified face candidate display field 350 instead of the post-conversion display field 330 of the conversion screen 300 of Fig. 6. The modified face candidate display field 350 includes a plurality of different modified faces after conversion of the person who is the conversion target 40. The operator can select one of the candidates as the modified face after conversion by performing a selection operation such as clicking on the image of the modified face. However, by providing a UI such as a radio button for each modified face candidate, the output processing unit 106 may accept a selection operation of the UI and accept the selection of the modified face after conversion from among the candidates.

[0085] The conversion screen 300 may display a selection notification field 362 in association with the selected modified face. The selection notification field 362 may display a message indicating that the face has been selected (e.g., "Selected"), but is not limited to this. For example, to indicate that the face has been selected, the display frame of the modified image 50 may be highlighted by changing it to a different color from the other display frames, or an image element such as a label or icon, such as "Selected," may be superimposed on the modified image 50.

[0086] Furthermore, after selecting the post-conversion modified faces from among multiple candidates for each of the multiple conversion targets 40 on the conversion screen 300 of FIG. 9, the selected modified faces may be displayed in a list in one-to-one association with the setting screen 400, as on the conversion screen 300 of FIG. 6.

[0087] According to this configuration, when there are a plurality of conversion objects 40, the post-conversion corrected image 50 can be selected from the candidates while checking harmony with the other conversion objects 40, so that a second image 22 that looks less unnatural can be generated.

[0088] <Reproduction Process of Second Image 22> When reproducing the second image 22, the output processing unit 106 may identifiably display the modified face of the person to be converted. For example, the modified face (modified image 50) that has been subjected to the conversion process may be superimposed with an associated image element such as a frame, label, or icon indicating that the conversion process has been performed. The operator may be able to select whether to display or hide the superimposed display of this image element.

[0089] Furthermore, while the display is being performed, the output processing unit 106 may accept selection of the correction image 50 and display the conversion screen 300. With this configuration, if the operator checks the converted correction image 50 on the reproduced image and feels something is wrong, he or she can return to the conversion screen 300, redo the conversion settings, and update the correction image 50. Furthermore, by viewing the reproduced image, the operator can check objects that are conversion targets 40 but have not been converted. In this case, the output processing unit 106 may accept designation of the area of ​​the conversion target 40 by a selection operation by the operator on the reproduced image, and display the conversion screen 300 for performing conversion processing on the conversion target 40. This allows conversion processing to be performed on objects that were not detected as conversion targets 40.

[0090] <Conversion Processing During Live Broadcasting and Live Streaming> In the above embodiment, a configuration for converting captured video has been described. In another embodiment, the image processing device 100 has a configuration for converting images captured by a camera 5 in real time. As shown in Fig. 10, the image processing system 1 further includes a camera 5 in addition to the configuration of Fig. 4. Here, processing in real time means sequentially acquiring and processing images generated by the camera 5, but also includes processing images delayed by a predetermined time, and processing images at predetermined time intervals together.

[0091] The camera 5 includes an imaging element such as a lens and a CCD (Charge Coupled Device) image sensor, and is, for example, a network camera such as an IP (Internet Protocol) camera. The network camera has, for example, a wireless LAN (Local Area Network) communication function, and is connected to the image processing device 100 via a relay device (not shown) such as a router of the communication network. The camera 5 may also include a mechanism for controlling the orientation of the camera body and lens, zooming, focusing, etc., in accordance with the movement of a person.

[0092] It is preferable that the images generated by the camera 5 are captured in real time and transmitted to the image processing device 100. However, the images transmitted to the image processing device 100 do not have to be transmitted directly from the camera 5, but may be images delayed by a predetermined time. The images captured by the camera 5 may be temporarily stored in another storage device, and the image processing device 100 may read them from the storage device sequentially or at predetermined intervals. Furthermore, the images transmitted to the image processing device 100 are moving images. However, the images transmitted to the image processing device 100 may be frame images transmitted at predetermined intervals, or may be still images.

[0093] As described above, the image processing device 100 may be realized in a form in which it is built into the camera 5 .

[0094] <Audio Conversion Processing> In the above embodiment, the image processing device 100 is configured to perform conversion processing on an image of the conversion target 40 in the video. In another embodiment, the image processing device 100 may be configured to perform conversion processing on audio in the video other than the audio of the shooting target. The shooting target storage unit 130 stores characteristic information about the audio of the person being filmed as shooting target information. Here, the characteristic information about the audio includes at least one of, for example, voice pitch, gender of the voice, age of the voice, speaking rate, presence or absence of stuttering, and voiceprint. The detection unit 104 then extracts audio from the video and performs a matching process with the shooting target information. Audio that matches the shooting target information to a threshold or higher is identified as the audio of the shooting target 30, and audio that matches the shooting target information to a threshold or lower is identified as the audio of the conversion target 40. The output processing unit 106 performs conversion processing on the audio of the conversion target 40 and outputs it as the audio of the second image 22. The conversion process may be, for example, a general voice changer process, or may include processes such as changing the pitch of the voice, the gender, the age, or the speed of speech. Furthermore, the detection unit 104 may detect abusive language or inappropriate words uttered by the conversion target 40 through voice recognition processing, and the output processing unit 106 may perform processing such as deleting the voice.

[0095] <Conversion Process Using Machine Learning Model> The output processing unit 106 may input specified attributes into a machine-learned model and generate and output the second image 22 including a modified face image (modified image 50) of the conversion target 40 output from the model. Here, the modified face data used to train the machine-learned model is a facial image of a fictional character. The attributes input to the model are attributes extracted from the conversion target 40 (gender, age, appearance, clothing, behavior), and do not include an image of the conversion target 40. Furthermore, the input attributes may include the facial expression of the conversion target 40 extracted from the conversion target 40. The facial expression specification may include a specification of whether to reflect the pre-conversion facial expression or a specification of whether to convert the pre-conversion facial expression to another facial expression. In the latter case, if a specification of conversion to another facial expression is received, a modified image 50 may be generated in which the crying face of the conversion target 40 before conversion is converted to a smiling face.

[0096] In this configuration, because the second image 22 that has been automatically converted is generated, it is preferable for an operator to confirm the second image 22 through the playback process of the second image 22 described above. Furthermore, during playback for confirmation, the second image 22 may be displayed in a manner that allows distinguishing between the corrected image 50 that has been automatically converted by the machine learning model, the corrected image 50 that has been converted using the conversion settings made by the operator, and an object that has not been subjected to the conversion process. In this case, as with the playback screen described above, it is preferable for the configuration to accept selection of an object for which the conversion process is to be redone and allow the conversion process to be redone.

[0097] <Confirmation Process of Deleted Target> On the playback screen described above, the output processing unit 106 may be able to display, in response to an operator's operation, an identification display (such as surrounding the area with a frame or superimposing an image element such as a mark or icon) that allows the area of ​​a hidden or deleted target to be confirmed. Alternatively, the output processing unit 106 may be able to re-display a hidden or deleted target in response to an operator's operation.

[0098] Furthermore, if the hidden or deleted object is a person, the output processing unit 106 may display a list of all objects detected from the first image 20, including the person. The list may also display results of narrowing down the objects by category. The categories include, for example, people in the photographed object 30, people in the conversion object 40 after conversion processing, people in the conversion object 40 that have not been changed, and people that have been hidden (deleted).

[0099] Furthermore, the image processing device 100 may omit the conversion process for a person that is recognized as a person among the objects extracted from the first image 20 but does not contain characteristic information sufficient to identify the individual. In this case, the output processing unit 106 may output the person on the playback screen so that the person can be identified, for example, by superimposing a gray hatch on the image. Furthermore, the playback screen may be capable of accepting selection of the person, transitioning to the conversion screen 300, and performing the conversion process on the person.

[0100] Another example of the present disclosure may be a program causing at least one computer to execute the method disclosed above, or a computer-readable recording medium having such a program recorded thereon. This recording medium includes a non-transitory tangible medium. The computer program includes computer program code that, when executed by a computer, causes the computer to perform the image processing method on an image processing device.

[0101] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present disclosure.

[0102] Furthermore, the various components of the present disclosure do not necessarily have to be independent entities, but may be formed as a single member by multiple components, one component may be formed from multiple components, one component may be a part of another component, or part of one component may overlap with part of another component, etc.

[0103] Furthermore, although the methods and computer programs disclosed herein describe a number of steps in a sequential order, the order in which the steps are performed does not limit the order in which the steps are performed. Therefore, when implementing the methods and computer programs disclosed herein, the order of the steps can be changed as long as it does not cause any problems in terms of the content.

[0104] Furthermore, the steps of the method and computer program disclosed herein are not limited to being executed at different times, and may be executed while one step is being executed, or may be executed partially or entirely overlapping with another step, etc.

[0105] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that would be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. Note that, in the present disclosure, when information about a target (especially the conversion target 40) is acquired and used, this must be done lawfully.

[0106] In addition, although the flowcharts used in the above description show a sequence of steps (processes), the order of steps executed in each embodiment is not limited to the sequence shown in the flowcharts. In each embodiment, the order of steps shown in the diagrams can be changed as long as it does not cause any problems in terms of the content.

[0107] Some or all of the above embodiments can be described as, but are not limited to, the following notes. 1. An image processing device comprising: an acquisition means for acquiring subject information related to a subject to be photographed; a detection means for processing a first image to detect at least one conversion subject other than the subject to be photographed identified using the subject information; and an output processing means for generating and outputting a second image after converting the detected conversion subject into another subject using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion subject. 2. An image processing device described in 1., wherein the conversion subject is a face of a person other than the subject to be photographed. 3. An image processing device described in 1. or 2., wherein the attributes of the conversion subject include at least one of the gender, age, appearance, clothing, and behavior of the person to be converted. 4. An image processing device described in 1. to 3. 5. An image processing device according to any one of items 1. to 4., wherein the output processing means outputs the second image in which information about the facial expression of the person before conversion is reflected in the modified face after conversion of the person to be converted. 5. An image processing device according to any one of items 1. to 4., wherein the output processing means inputs specified attributes into a machine-learned model, and generates and outputs the second image including an image of the modified face of the person to be converted output from the model. 6. An image processing device according to any one of items 1. to 5., wherein the output processing means displays the modified face of the person to be converted in an identifiable manner when playing back the second image. 7. An image processing device according to any one of items 3. to 6. which cites items 2. and 2., wherein the output processing means displays the image of the face of the person to be converted and the image of the modified face of the person after conversion side by side on the same screen before generating the second image.8. An image processing device according to claim 7, wherein the conversion rule includes at least one setting item, and the screen further includes a receiving means associated with the conversion target for receiving setting input of the at least one setting item according to an attribute of the person who is the conversion target, and the output processing means, upon receiving the setting input by the receiving means associated with the conversion target, updates the modified face of the person who is the conversion target using the conversion rule in accordance with the entered setting item, and displays the updated face on the screen. 9. An image processing device according to claim 7 or 8, wherein the output processing means groups faces of the multiple people who are the conversion targets using a predetermined criterion, and displays images of the modified faces of the multiple people after converting the conversion targets on a group-by-group basis, using the conversion rule set for each group. 10. An image processing device according to claims 3 to 9, which cite 2 and 2. wherein the output processing means displays a plurality of different modified faces of the person to be converted on the same screen, accepts selection of one of the plurality of modified faces, and outputs the second image in which the selected modified face is the modified face of the person to be converted.

[0108] 11. An image processing method in which one or more computers acquire subject information related to a subject to be photographed, process a first image to detect at least one conversion subject other than the subject to be photographed identified using the subject information, and generate and output a second image after converting the detected conversion subject into another subject using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion subject. 12. A program that causes a computer to execute an acquisition process of acquiring subject information related to a subject to be photographed, a detection process of processing a first image to detect at least one conversion subject other than the subject to be photographed identified using the subject information, and an output process of generating and output a second image after converting the detected conversion subject into another subject using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion subject. 13. A computer-readable recording medium storing a program that causes a computer to execute: an acquisition process for acquiring object information relating to an object to be photographed; a detection process for detecting at least one conversion object other than the object to be photographed identified using the object information by processing a first image; and an output process for generating and outputting a second image after converting the detected conversion object into another object using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion object.

[0109] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 10 that are subordinate to Supplementary Note 1 (image processing device) may also be subordinate to Supplementary Note 11 (image processing method), Supplementary Note 12 (program), and Supplementary Note 13 (recording medium) in the same subordinate relationship as Supplementary Note 2 to 10. Furthermore, not limited to Supplementary Note 11, Supplementary Note 12, and Supplementary Note 13, some or all of the configurations described as Supplements may be subordinate to various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0110] This application claims priority based on Japanese Patent Application No. 2024-025651, filed February 22, 2024, the disclosure of which is incorporated herein by reference in its entirety.

[0111] REFERENCE SIGNS LIST 1 Image processing system 5 Camera 20 First image 22 Second image 30 Photographed object 40, 40a, 40b Conversion object 50, 50a, 50b Corrected image 100 Image processing device 102 Acquisition unit 104 Detection unit 106 Output processing unit 120 Storage device 130 Photographed object storage unit 140 Video storage unit 150 Conversion image storage unit 300 Conversion screen 400 Setting screen 500 Attribute setting screen 1000 Computer 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input / output interface 1060 Network interface

Claims

1. An image processing device comprising: an acquisition means for acquiring object information relating to an object to be photographed; a detection means for detecting at least one conversion object other than the object to be photographed identified using the object information by processing a first image; and an output processing means for generating and outputting a second image after converting the detected conversion object into another object using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion object.

2. An image processing device according to claim 1, wherein the object to be converted is a face of a person other than the subject of the image capture.

3. An image processing device according to claim 1 or 2, wherein the attributes of the person to be converted include at least one of the gender, age, appearance, clothing, and behavior of the person to be converted.

4. An image processing device according to any one of claims 1 to 3, wherein the output processing means outputs the second image in which information about the facial expression of the person to be converted before conversion is reflected in the modified face of the person to be converted after conversion.

5. An image processing device according to any one of claims 1 to 4, wherein the output processing means inputs the specified attributes into a machine-learned model, and generates and outputs the second image including the image of the modified face to be converted output from the model.

6. An image processing device according to any one of claims 1 to 5, wherein the output processing means, when reproducing the second image, displays the modified face of the person to be converted in a identifiable manner after conversion.

7. An image processing device as claimed in claim 2 and any one of claims 3 to 6 which cite claim 2, wherein the output processing means displays, before generating the second image, an image of the face of the person to be converted and an image of the modified face of the person after converting the object to be converted side by side on the same screen.

8. An image processing device according to claim 2 and any one of claims 3 to 7 which cite claim 2, wherein the output processing means displays a plurality of different modified faces of the person to be converted on the same screen, accepts the selection of one of the plurality of modified faces, and outputs the second image in which the selected modified face is the modified face of the person to be converted.

9. An image processing device according to claim 7, wherein the conversion rule includes at least one setting item, and the screen further includes a receiving means associated with the conversion target for receiving setting input of the at least one setting item according to an attribute of the person who is the conversion target, and the output processing means, upon receiving the setting input by the receiving means associated with the conversion target, updates the modified face of the person who is the conversion target using the conversion rule in accordance with the setting item that has been set and input, and displays it on the screen.

10. An image processing device according to claim 7 or 9, wherein the output processing means groups the faces of the plurality of people to be converted using a predetermined criterion, and displays on the screen images of the modified faces of the plurality of people after converting the objects to be converted on a group-by-group basis using the conversion rule set on a group-by-group basis.

11. An image processing method in which one or more computers acquire object information relating to an object to be photographed, process a first image to detect at least one conversion object other than the object to be photographed identified using the object information, and generate and output a second image in which the detected conversion object is converted into another object using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion object.

12. A computer-readable recording medium storing a program that causes a computer to execute: an acquisition process for acquiring object information relating to an object to be photographed; a detection process for detecting at least one conversion object other than the object to be photographed identified using the object information by processing a first image; and an output process for generating and outputting a second image after converting the detected conversion object into another object using a conversion rule, wherein the conversion rule differs depending on at least some of the attributes of the conversion object.

Images