Image display device and image display method

Abstract

To display a real image of a specific part of a person in an image while taking into consideration the privacy of the person in the image. [Solution] The image display device 100 is configured to include a storage device 101 that stores captured images including real images of the subject, and a processor 105 that executes processes to extract real images that include specific parts of the subject from the real images, to make areas in the captured images other than the specific parts non-real, and to synthesize and display the real images that include the specific parts with the captured images that have been made non-real.

Description

[Technical Field]

[0001] The present invention generally relates to techniques for image display devices and image display methods, and more particularly to a technique that enables the display of a realistic image of a specific part of a person in an image while taking into consideration the privacy of the person in the image. [Background technology]

[0002] Thanks to the dramatic improvements in the capabilities and size reduction of computer chips, VR (Virtual Reality) and AR (Augmented Reality) implementation technologies are becoming applicable to business. For example, there is technology that creates digital twins by synthesizing avatars or real images of workers or passersby in a virtual space that is based on the real world.

[0003] However, when controlling such images, if actual images of the people involved are used without restriction, problems may arise in terms of privacy protection and psychological stress. Patent Document 1, therefore, has been proposed as a conventional image processing technology that takes these problems into consideration. Patent Document 1 discloses a technology that protects privacy without obstructing images captured within a set area, and allows many people to view the image of the area efficiently and quickly.

[0004] Furthermore, a technology has been disclosed that provides a sensor system or the like with improved availability by avoiding issues related to the protection of personal information such as portrait rights (see Patent Document 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-150849 [Patent Document 2] Japanese Patent Publication No. 2022-117761 Summary of the Invention [Problem to be solved by the invention]

[0006] However, if the entire person in the video is masked, as in conventional technology, meaningless video may be generated and displayed depending on the intended use of the video. For example, if the video is to provide educational content showing the work of a skilled craftsman, it is essential that the person's detailed movements can be confirmed in the video. In other words, if control technology that masks the entire person is used, it is difficult to generate and provide useful video. On the other hand, from the perspective of privacy protection, there is still a need for control that appropriately masks specific parts (or the entire person) of a person depending on the situation.

[0007] Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that enables the display of realistic images of specific parts of a person in an image while taking into consideration the privacy of the person in the image. [Means for solving the problem]

[0008] The image display device extracts a real image that includes a specific part of the subject from the real image of the subject, converts areas other than the specific part in the photographed image that includes the real image of the subject into a non-real image, and combines the real image that includes the specific part with the photographed image that has been non-realized and displays the image. [Effects of the Invention]

[0009] According to the present invention, it is possible to display a realistic image of a specific part of a person in an image while taking into consideration the privacy of the person in the image. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a network configuration including an image display device according to an embodiment. [Figure 2] 1 is a diagram illustrating an example of the configuration of an image display device according to an embodiment. [Figure 3] FIG. 2 illustrates an example of the configuration of a management terminal according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a user terminal according to an embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of the configuration of user data in the embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of work video data according to an embodiment. [Figure 7] FIG. 4 is a diagram illustrating an example of the configuration of work position data according to the embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of the configuration of background image data in the embodiment. [Figure 9] FIG. 4 is a diagram illustrating an example of the configuration of display condition data according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the flow of an image display method according to an embodiment. [Figure 11] FIG. 10 is a diagram illustrating a conceptual example of identifying a work area in the embodiment. [Figure 12] 10A and 10B are diagrams illustrating an example of a display control concept based on a paste code in an embodiment. [Figure 13A] FIG. 10 is a diagram illustrating a conceptual example of background image generation in the embodiment. [Figure 13B] FIG. 10 is a diagram illustrating a conceptual example of background image generation in the embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of the flow of an image display method according to an embodiment. [Figure 15] FIG. 1 is a diagram illustrating a conceptual example of privacy protection control in an embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of a display control concept according to user attributes in an embodiment. [Figure 17] 10A and 10B are diagrams illustrating an example of a concept of displaying an image of a blind spot in an embodiment. [Figure 18] 10A and 10B are diagrams illustrating examples of suggested mask areas designated by an administrator in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following description, a communication device may be one or more communication interface devices, which may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., a NIC and an HBA (Host Bus Adapter)).

[0012] In the following description, a "memory" refers to one or more memory devices, which are an example of one or more storage devices. At least one of the memory devices may be a volatile memory device or a non-volatile memory device.

[0013] In the following description, a "storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device, specifically, for example, a hard disk drive (HDD), a solid state drive (SSD), or a non-volatile memory express (NVMe) drive.

[0014] In the following description, a processor is an arithmetic unit and may be one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a hardware circuit that performs part or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0015] In the following description, information that provides an output in response to an input may be described using expressions such as "xxx table" or "xxx database." However, this information may be data of any structure (for example, structured data or unstructured data), or may be a learning model such as a neural network, genetic algorithm, or random forest that generates an output in response to an input. Therefore, "xxx table" or "xxx database" may be referred to as "xxx information." In the following description, the structure of each database or table is an example, and one database or table may be divided into two or more databases or tables, or all or part of two or more databases or tables may be one database or table.

[0016] In the following description, processing may be described using a "program" as the subject. However, because a program is executed by a processor to perform a predetermined process using a storage device and / or an interface device, etc., as appropriate, the subject of the process may also be the processor (or a device such as a controller having the processor). A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] In addition, in the following description, when describing elements of the same type without distinguishing between them, common parts of the reference symbols may be used, and when describing elements of the same type with distinction between them, reference symbols or element identifiers may be used. <Network configuration including image display devices>

[0018] FIG. 1 is a diagram showing an example of a network configuration of an image display device 100 according to an embodiment of the present invention.

[0019] The image display device 100 of this embodiment is a device that allows display of a realistic image of a specific part of a person in the image while taking into consideration the privacy of the person in the image, and is connected to a management terminal 200, a user terminal 300, and an imaging system 400 via a network N. Therefore, these may be referred to as an image display system 10.

[0020] The image display device 100, which is the main component of the image display system 10, stores photographed images including real images of the subject in a storage device, and performs the process of extracting real images including specific parts of the subject from the real images, converting areas of the photographed images other than the specific parts into non-real images, and synthesizing the real images including the specific parts with the photographed images that have been non-real images and displaying them.

[0021] Furthermore, the image display device 100 constituting the image display system 10 is communicably connected to a management terminal 200 via an appropriate network N. This management terminal 200 is, for example, a terminal used by a manager of a production line in a factory. The manager, who is the user of this management terminal 200, connects the management terminal 200 to the image display device 100 via the network N and wishes to provide (using the image display device 100) actual images that are useful for understanding the work content of skilled workers to the user terminals 300 of the workers working on the production line (which may conceptually include workers in various environments, such as those actually working on site and those learning about the work content of skilled workers at a predetermined training facility, etc.).

[0022] The user terminal 300 is, for example, a terminal used by a worker working on a production line in a factory. The user terminal 300 is connected to the image display device 100 via the network N, and receives and displays images of the work being done by a skilled worker, particularly images capturing the worker's hands, while the worker is working on the production line or undergoing training at a training facility or the like.

[0023] Examples of implementations of this user terminal 300 include a notebook PC, a tablet terminal, and a first-person perspective wearable camera such as VR goggles or an AR camera. When such a wearable camera is used as the user terminal 300, the user, i.e., the worker, can view a video of the work content at the work site (actual images of the work content of the skilled worker) from his / her own perspective.

[0024] Furthermore, the photographing system 400 is, for example, a system that photographs the above-mentioned factory production line, including the workers, and provides the photographed images to the image display device 100. This photographing system 400 is a system composed of multiple digital video cameras that are placed, for example, directly above the production line or at various locations around it, and that photograph the hands of the workers performing the work and the objects being worked on at a certain size. The photographing system 400 can also be considered an image server device that transmits data of the photographed images taken by each digital video camera to the image display device 100 via the network N, along with meta information such as the identification information and position of the production line being photographed and the date and time of the photograph. Each digital video camera in such photographing system 400 may include a wearable camera, which is the user terminal 300.

[0025] The image display device 100 shown in Figures 1 and 2 includes a storage device 101 consisting of a non-volatile storage device such as a hard disk drive or an embedded multimedia card, an I / O 102 for connecting to input devices such as a keyboard and output devices such as a display, a memory 103 consisting of a volatile storage device such as a RAM (Random Access Memory), a communication device 104 connected to a network N and communicating with at least an administration terminal 200, a user terminal 300, and an imaging system 400, and a processor 105 that calls a program 1010 stored in the storage device 101 into the memory 103 to perform overall control of itself and perform various judgments, calculations, and control processing.

[0026] The functions implemented in the storage device 101 of the image display device 100 include a work video capturing unit 110, a person recognition unit 111, a display part identification unit 112, a background image generation unit 113, a display video generation unit 114, and a video display unit 115 (see FIG. 1). Details of each of these functions will be described later. The storage device 101 also holds at least an AR engine 1011. The AR engine 1011 is an engine that generates display data for a so-called augmented reality space by combining an image of real space captured by a wearable camera, which is the user terminal 300, with a corresponding separate image (such as an object, icon, text, or figure from a generated / captured image). Of course, a VR engine may be provided instead of or in addition to the AR engine 1011. The VR engine is an engine that generates display data for a virtual reality space.

[0027] Note that part of the processing performed by the processor 105 when it executes the program 1010 may be executed by another arithmetic device (for example, hardware such as an ASIC or FPGA). The memory 103 may be a ROM (Read Only Memory), which is a non-volatile storage element. The ROM stores unchanging programs (for example, BIOS). The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores the program 1010 executed by the processor 105 and data used when the program 1010 is executed.

[0028] Furthermore, the above input and output devices may be provided by a management terminal 200 or a user terminal 300 connected to the image display device 100 via a network N. The program 1010 executed by the processor 105 is provided to the image display device 100 via removable media (CD-ROM, flash memory, etc.) or the network N, and is stored in a non-volatile storage device 101, which is a non-transitory storage medium.

[0029] Therefore, the image display device 100 reads data from the removable media via the I / O 102. The image display device 100 is a computer system configured on one physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. The image display device 100 is preferably configured on a cloud, but may also be on-premise configured on a specific computer (hardware).

[0030] The network N connecting the image display device 100 with the management terminal 200, the user terminal 300, and the photography system 400 may be the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or a mobile phone network, but is not limited to these.

[0031] The management terminal 200 connected to the network N is specifically implemented in the form of a PC (Personal Computer), a tablet terminal, a smartphone, etc. Similar to the image display device 100, the management terminal 200 includes a storage device 201, an I / O 202, a memory 203, a communication device 204, and a processor 205.

[0032] As already mentioned, the user terminal 300 is, for example, a first-person perspective wearable camera. Similar to the image display device 100, the user terminal 300 includes a storage device 301, an I / O 302, a memory 303, a communication device 304, and a processor 305 (see FIG. 3). Of these, the storage device 301 stores, in addition to a program 3011, an AR application 3012, for example. The AR application 3012 is an application that displays (plays back) on the display of the wearable camera, the display data of the augmented reality space that has been generated and distributed by the AR engine 1011 of the image display device 100.

[0033] Note that data exchange between the image display device 100 and the management terminal 200, the user terminal 300, and the imaging system 400 may be performed according to, for example, an API (Application Programming Interface) protocol. In this case, it is assumed that each device is pre-implemented with the functions and configurations for executing each process of request and response by the API. <Example of data configuration>

[0034] Next, specific examples of data held and managed by the image display device 100 will be described with reference to FIGS.

[0035] 5 is a diagram showing an example of the configuration of user data 1013 in this embodiment. This user data 1013 is a table that stores the values ​​of each user's account name, attribute, user not requiring masking, and user not requiring disclosure. Note that "user" includes, for example, the manager of the above-mentioned production line and the workers working on the production line.

[0036] Of the above values, the account name is information such as the user's name and employee number. The attributes are information indicating the user's job position and skills. The mask-free users are information indicating people (other users) to whom the user may disclose actual images (including specific body parts) of themselves as the subject. The non-disclosure users are information indicating people (other users) to whom the user does not want to disclose actual images (including specific body parts) of themselves as the subject, in other words, people to whom the user wants to keep secret.

[0037] Next, the work video data 1014 of this embodiment will be described. Fig. 6 is a diagram showing an example of the configuration of the work video data 1014 of this embodiment. The work video data 1014 indicates information about captured images including workers working on the production line, and is a table storing values ​​such as the video ID, work location, camera type, work content, shooting date and time, worker, and video file name.

[0038] Of the above values, the video ID is identification information for each captured image. The work location is identification information for the production line that is the subject of the image capture. The camera type is information indicating whether the camera is a fixed camera installed around the production line or a wearable camera worn by a worker. The work content is information indicating the work content on the production line. This work content is, for example, information regarding the work content for each production line obtained from a management system such as a work management system. The image capture date and time is information indicating the date and time the image capture was performed.

[0039] The worker is the account name of the worker who received a task assignment and performed work on the production line. This worker information is also set as information about the worker for each production line obtained from a management system such as a work management system. Note that the numbers attached to the account name as worker information indicate the coordinates of the person's head in the first frame image in which the person appears during the shoot. For example, by registering information such as "Person A is located near coordinates X and Y" based on the work position in advance, the person and coordinates can be associated. The coordinates of such a person's position are obtained based on the person's skeletal information, and the same person in other frames can be identified by tracking the coordinates of the same person in other frames based on the amount of change in coordinates between previous and next frames (person + coordinate information can also be managed for each frame). Note that if there is only one worker, the person can be identified without managing coordinates, so there is no need to identify the person or manage coordinates. The video file name is information indicating the file name in which the data of the captured image is stored.

[0040] Next, the working part data 1015 of this embodiment will be described. FIG. 7 is a diagram showing an example of the configuration of the working part data 1015 of this embodiment. The working part data 1015 is a table storing data of actual images including a specific part (e.g., hands) of each worker, extracted from the captured image. The specific part, i.e., the working part, is a predetermined (specified) part among parts identified by image analysis (details omitted as this is a known technique) based on skeletal information of the subject in the captured image. Here, the hand is considered to be the specific part. In other words, an image corresponding to an area of ​​a certain radius including the subject in the captured image, i.e., the worker's hand, is extracted from the captured image, and this data becomes the working part data.

[0041] As shown in FIG. 7, the work part data 1015 is composed of the values ​​of video ID, start time, end position, worker, and work part video data. Of these, the video ID is identification information that uniquely identifies the work part video data. The start time is information indicating the start time of the work part video data. The end position is information indicating the end time of the work part video data. The worker is the account name of the worker who appears as a subject in the work part video data. Furthermore, the work part video data is a value indicating the file name of the work part video data.

[0042] Note that the work part data 1015 must be registered for each work location, and preparing a video of the same length as the work video data for each specific part would result in an excessive amount of data. Therefore, in order to save only the time when work is being performed (excluding time when no one is present), the above start time and end time for the work video data are registered. Note that the work video data is an image in which the background image can be separated (for example, of the area of ​​the background image and the area of ​​the actual image that includes the specific part, the background image area can be set to a transparent color or a color not used for the specific part, so that it is possible to identify which specific part in the image it is), making it easy to see where in the background video the work part should be composited.

[0043] Next, the background image data 1016 of this embodiment will be described. FIG. 8 is a diagram showing an example of the configuration of the background image data 1016 in this embodiment. The background image data 1016 is image data of an area other than the actual image including a specific part in the captured image. Therefore, the background image data 1016 is a table including values ​​such as a video ID, a background image generation means, and a background image file name. Of these, the video ID is identification information that uniquely identifies the background image data. The background image generation means indicates the type of background image generation means, such as generation AI, overall mosaic processing, blacking out human-detected parts, or blacking out difference parts identified by difference analysis, which are the processing details of the background image generation unit 113. Furthermore, the background image file name is identification information that uniquely identifies the work part data extracted from the captured image.

[0044] Note that background images are primarily video, as they may include moving subjects such as conveyor belts on a production line. However, if fixed images are permitted for the subject other than the work area (specific area), still images may also be used. In the case of video, by making the video length the same as the work video, it becomes possible to match the work video data with the background image data using the video offset time.

[0045] Furthermore, among the background image generation means, "blacking out detected person parts" corresponds to masking (an example of creating a non-realistic image) an image area identified as a person among the subjects by skeleton or person detection with a specific color such as black (this concept may also include a wider masking that extends out to a certain percentage from the image area of ​​the person). Note that the image area of ​​the person can also be identified on a pixel-by-pixel basis by employing a publicly known technology such as instance segmentation.

[0046] Additionally, among background image generation means, "whole mosaic" corresponds to a method that performs mosaic on the entire photographed image or on each image area of ​​each object or person while preserving the outlines of the object. On the other hand, among background image generation means, "generation AI" corresponds to a method that generates and replaces at least the people in the photographed image with similar images of fictional characters by inputting a photographed image of a production line into the generation AI.

[0047] Furthermore, among the background image generation means, "difference analysis" identifies areas where there are differences between previous and subsequent image frames in images taken by fixed cameras (digital video cameras that make up the photography system 400) installed around the production line (however, moving parts such as conveyor belts are excluded from the difference areas by registering their locations in advance, even if there is movement), and identifies those areas as people (or parts of people).

[0048] Next, the display condition data 1017 of this embodiment will be described. FIG. 9 is a diagram showing an example of the configuration of the display condition data 1017 in this embodiment. The display condition data 1017 is information that specifies whether or not to display a real image of a person as a subject, a specific part of the person, or a surrounding area thereof. In the example shown in FIG. 9, data that associates values ​​of display and non-display (non-real image) that are actions related to a real image with conditions for adopting the action is listed. The image display device 100 controls to display a real image or non-real image (non-real image) for an image area of ​​a subject in a captured image that is recognized as a person through image recognition, skeletal analysis, or the like, in a situation that corresponds to each condition of the display condition data 1017. <Image display method: Main flow>

[0049] Next, the processing flow of the image display method of this embodiment will be described together with each functional unit in the image display device 100.

[0050] 10 is a diagram showing an example of the flow of the image display method in this embodiment. In explaining this example of the flow, each functional unit of the image display device 100 shown in FIG.

[0051] First, the work video shooting unit 110 of the image display device 100 works in cooperation with the photography system 400 or the worker's user terminal 300 (which is a wearable camera) to shoot work video (shot image) of a predetermined area including the production line 1 to be shot, as shown in FIG. 11 (S1). In other words, the work video shooting unit 1110 is a functional unit that works in cooperation with the photography system 400 or the user terminal 300 to acquire work video of the target. Therefore, the work video shooting unit 110 controls, for example, the shooting control functions of the photography system 400 or the wearable camera (e.g., start / end of shooting, change of angle of view, photography position, direction, selection, or selection of a digital video camera) as needed via the network N. The work video shooting unit 110 stores the acquired work video in work video data 1014.

[0052] Furthermore, the display part identification unit 112 of the image display device 100 provides the above-mentioned work video to the person recognition unit 111, which identifies an image area relating to the worker (person) among the subjects in the work video, and identifies the work part 4 (see FIG. 11) based on the information on the image area (S2). The person recognition unit 111 identifies the image area of ​​the worker by appropriately employing known person detection technology (including AI-based pattern recognition, etc.). Furthermore, as already mentioned, in identifying the work part 4, by appropriately employing existing technology such as skeletal analysis, a predetermined part (e.g., hand) or a part showing a specific movement (e.g., a part that moves back and forth more than a standard amount within a certain period of time) is identified from among parts of the human body divided into joints (e.g., head (face), neck, shoulders, upper arms, forearms, hands, thighs, lower legs, feet, etc.).

[0053] Furthermore, the display part identification unit 112 of the image display device 100 registers the information and video data of the work part identified in S2 in the work part data 1015 (S3). What is registered here is a video of a certain range (e.g., within a predetermined distance from the center of the pixel group of the work part 4, including the work object 2 such as a part) including the work part 4 (e.g., hand) of a worker working on the production line 1, as shown in Figs. 7 and 11, that is, a work part video 5.

[0054] When identifying and registering the work parts, it is also possible to identify and register parts of the worker's body parts that have been predetermined as privacy protection targets (e.g., face). The registration of the privacy protection target parts may be performed by adding information indicating that the parts are privacy protection targets to the work part data 1015, or may be performed in a separately generated table.

[0055] Furthermore, the display part identification unit 112 may employ a method for identifying the work part 4 by reading a two-dimensional code 7 that is actually attached (printed or affixed) to the work part 4 of the worker (see FIG. 12), instead of (or as a supplement to) the method for identifying the work part of the subject in the work video by skeletal analysis or the like as described above. As shown in FIG. 12, the two-dimensional code 7 may be affixed to the work part 4, such as the worker's hand, or may also be attached to the work object 2 that should be included in the work part video. Furthermore, the two-dimensional code 7 may be attached to a concealed part 6 that should not be displayed, and by reading this two-dimensional code 7, the part to which the code is attached may be recognized as a concealed part.

[0056] Such two-dimensional code 7 is an encoded identification value that indicates whether it is a work area or a secret area. In this case, the display area identification unit 112 reads the above value by applying a two-dimensional code analysis application or the like (held by the image display device 100 or called from an external service via the network N and used) to the image of the two-dimensional code 7 included in the work video captured by the digital video camera of the imaging system 400.

[0057] Furthermore, the display part identification unit 112 may identify the work part by, for example, accepting a designation of the work part in the work video from the management terminal 200 or the user terminal 300. In this case, the display part identification unit 112 delivers the work video to the management terminal 200 or the user terminal 300, and accepts a designation operation on an appropriate user interface there, thereby identifying a pixel area including the location of the designation operation as the work part. Such designation of the work part is made by at least one of the manager and the worker.

[0058] Furthermore, such designation operations within the work video may be performed by specific operations such as moving, touching, or pointing at a cursor or avatar within the work video displayed as augmented reality or virtual reality on the administrator's or worker's VR / AR goggles (corresponding to or attached to the management terminal 200 or user terminal 300).

[0059] Next, the background image generation unit 113 of the image display device 100 generates a background image to be composited with the work part image 5 registered in S3 above, based on the work image obtained in S1 above (S4). This background image generation process is performed using a predetermined background image generation means (human detection part blacking, overall mosaic, generation AI, and difference analysis) already described above, or a means selected by instruction from the management terminal 200. The background image generation unit 113 also stores the generated background image in background image data 1016 together with information about the background image generation means (S5).

[0060] Of these, background image generation by the generation AI involves inputting a prompt work video 12A1 to the generation AI along with an instruction to generate and replace, for example, person 12A2 among the subjects with fictional person 12A3, thereby generating background image 12A4, as shown in Figure 13A. Furthermore, background image generation using an overall mosaic involves performing a mosaic conversion on, for example, person 12B2 among the subjects in work video 12B1, thereby generating background image 12B3, as shown in Figure 13B.

[0061] Next, the image display device 100 determines whether to continue the series of processes that began with the shooting of the work video (S6). This determination is made based on, for example, a predetermined condition, such as whether the end time of the production line operation has arrived or whether an instruction to end shooting has been received from the management terminal 200. If the result of the above determination is to continue shooting the work video (S6: Y), the image display device 100 transitions the process to S1. On the other hand, if the result of the above determination is not to continue shooting the work video (S6: N), the image display device 100 ends this flow. <Image display method: Video display>

[0062] Next, a process flow for combining the video of the work area registered in the above flow with the background image generated in the above flow and displaying (playing) it appropriately on the user terminal 300 (or management terminal 200) will be described.

[0063] FIG. 14 is a diagram showing an example of the flow of an image display method according to this embodiment.

[0064] Here, the display image generation unit 114 of the image display device 100 acquires user information (S10). The user information is data stored in the user data 1013 that relates to those involved in the current image display, i.e., the viewers of the image and the subjects of the image. Therefore, the display image generation unit 114 is assumed to have acquired the identification information of such people (e.g., account names) from the management terminal 200 or the user terminal 300 during this processing. In the example shown in FIG. 5, information such as attributes, users not requiring masking, and users not permitted to disclose, which are linked to the target account name, is acquired.

[0065] Furthermore, the display video generation unit 114 of the image display device 100 determines the display timing based on the user information obtained in S10 (S11). This determination is made, for example, by determining the timing when a manager, worker, or their avatar arrives at or is located at a specific work position on the production line in an augmented reality or virtual reality space displayed on the management terminal 200 or the user terminal 300. Therefore, the display video generation unit 114 identifies a record in the work video data 1014 in which the user (viewer or subject) indicated by the user information in S10 above is a "worker," and identifies the value of the "video ID" (or "work location") indicated by that record. Note that the processing in S11 above may not be performed when real-time processing is performed on the work video to display an image including the work location.

[0066] Furthermore, the display image generation unit 114 of the image display device 100 acquires the corresponding images (working area image and its background image) from the working area data 1015 and the background image data 1016 based on the display timing determined in S11 above (S12). In this case, the display image generation unit 114 uses the image ID obtained in S11 as a key to identify and acquire the working area image data from the working area data 1015 and the background image data from the background image data 1016. Through this process, when the viewer or subject reaches a specific location in the space, the relevant image etc. is automatically acquired for the work being performed at that location and for which the image is registered in the working area data 1015 or the background image data 1016.

[0067] Furthermore, the video display unit 115 of the image display device 100 generates a display image by combining the video of the working part acquired in S12 with a background image (S13). This display image is configured such that only the working part, such as the worker's hands, is a real image, and the other areas are appropriately rendered non-realistic by blacking out, mosaic processing, etc.

[0068] In generating the display image, the video display unit 115 preferably selects, as the video to be used during generation, a work part video of a work part that can be displayed to the viewer (which may include the concept of a worker), or a work part video of a specific part of a specific subject, based on the job position and skill values ​​indicated by the "attributes" related to the viewer in the user data 1013 and the conditions indicated by the display condition data 1017 (display conditions related to at least one of the work part and the subject).

[0069] For example, based on a "condition" that a viewer of a certain job title can display work part images of "a specific worker A" or "a worker with skill rank A or higher," the "specific worker A" or "a worker with skill rank A or higher" is identified in the user data 1013, and the work part image is selected and composited into the background image. Alternatively, based on a "condition" that a viewer of a certain job title can display work part images of "at hand" for all workers, the work part image is identified in the work part data 1015 and composited into the background image. When such processing is performed, the images viewed by the viewer on the user terminal 300 will be different for each viewer (user), even if the viewer views the same production line 1 at the same time, as shown in FIG. 16.

[0070] Alternatively, for each user in the user data 1013 who is a target (viewer or worker) and does not require a mask, at least a work part image (a real image that has not been rendered as a fake image by mosaic processing or the like) may be selected from the work part data 1015. Also, for each user in the user data 1013 who is a target (viewer or worker) and for whom disclosure is not permitted, at least a work part image may be selected from the work part data 1015, and a fake image that has been rendered as a fake image by mosaic processing or the like may be selected, or the work part image itself may not be displayed regardless of the fake image processing.

[0071] As described above, the work part video selected according to the attributes of the viewer or the subject may be selected as an additional video to be placed around the video of the worker's hands (real-time video) on the wearable camera (user terminal 300) of the worker working on the actual production line. This operation can be applied to all work part videos (same applies hereinafter). Similarly, the control to prevent the display of the work part video of a user who is not permitted to disclose the video may be executed in the video displayed on the worker's wearable camera. In this case, the control may be performed in an augmented reality or virtual reality space.

[0072] When displaying the image of the work area in an augmented reality or virtual reality space, the image display unit 115 may perform an operation in which the control of the image of the work area or the non-realistic imaging (mask, etc.) of the entire subject is different for a worker working on the same production line and a manager who remotely accesses the image from a location other than the production line (e.g., a management office).

[0073] In this case, video display unit 115 identifies a group of subjects located in the space (or around the production line) based on the positions of the subjects in the space (acquired from user terminal 300). Video display unit 115 also acquires real images of each subject in the group of subjects from the wearable camera and selects them as images to be distributed to user terminal 300 of the person corresponding to each subject in the group of subjects (or the wearable camera displays them as they are without going through video display unit 115). Furthermore, when a viewer present in a different location outside the space (identified by acquiring location information from the viewer's terminal, management terminal 200, or user terminal 300) requests to view work site images related to the group of subjects in the space, video display unit 115 performs non-real image processing on each subject in the group of subjects and selects them as images to be distributed to the viewer's terminal.

[0074] Furthermore, the video display unit 115 may perform control such that, for example, when the hand R2, which is the work part 4, moves and overlaps or approaches the face area R1 (prohibited area) of the subject, the face area R1 is excluded from the display target (see FIG. 15). In this case, the video display unit 115 detects, through image recognition processing, that the pixel area of ​​the work part 4 approaches or overlaps the pixel area of ​​the face area R1 in the work video, and at that time erases the display of the face area R1, which is the prohibited area, or performs specific image control (e.g., mosaic processing, etc.).

[0075] Following S13 in the above flow, the video display unit 115 of the image display device 100 delivers the display image generated in S13 to the user terminal 300 (or the management terminal 200) via the network N and displays it (S14). At this time, depending on the positional relationship between the work object 2 and the work part (hand) on the production line 1, and the positional relationship between the shooting direction of the digital video camera of the shooting system 400 and the work part, there may be cases where only an image of the work part with insufficient accuracy can be delivered.

[0076] Therefore, the video display unit 115 determines whether the display area (number of pixels) of the work part in the work video is below a standard (e.g., an absolute standard, or the ratio of the number of pixels considered appropriate for the hand, the work part, to the number of pixels of the entire subject person, etc.), and if the display area is below the standard, it acquires work video (separately captured image) from a digital video camera of the imaging system 400A (see FIG. 17) that captures images of the production line 1 from a different shooting direction from the work video originally adopted. The video display unit 115 also extracts an actual image including the work part from this new separately captured image of the work video, and displays it in the work video in the form of, for example, a speech bubble G10.

[0077] Furthermore, the video display unit 115 may display a real image of the mask area (area to be rendered as a fictitious image) designated by the manager on the VR / AR goggles (user terminal 300) of the worker working on the production line 1, while superimposing a guide defining the mask area on the work video (see FIG. 18). The example in FIG. 18 shows a form in which the mask area is highlighted with a dashed line and a specific color, and a legend G11 is displayed. In this case, the worker can understand that the work part 4, such as the hand, and the work object 2 are areas designated by the manager as areas to be rendered as a fictitious image in the augmented reality or virtual reality space, and can perform the work.

[0078] Furthermore, the video display unit 115 of the image display device 100 determines whether to continue displaying the display image (S15). This determination is made based on conditions such as whether a predetermined viewing period for the display image has ended, whether the end time of the production line operation has arrived, or whether an instruction to end the display has been received from the management terminal 200. If the result of the above determination is to continue displaying the video (S15: Y), the video display unit 115 of the image display device 100 transitions the process to S10. On the other hand, if the result of the above determination is not to continue displaying the video (S15: N), the image display device 100 ends this flow.

[0079] As described above, the image display device of this embodiment makes it possible to display a realistic image of a specific part of a person in an image while taking into consideration the privacy of the person in the image.

[0080] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0081] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0082] The above various explanations can be summarized as follows: The following summary may include supplementary explanations and explanations of variations of the above explanations.

[0083] The image display device includes a storage device and a processor. The storage device stores photographed images including real images of a subject. The processor executes a process of extracting a real image including a specific part of the subject from the real images, a process of converting a region of the photographed image other than the specific part into a non-real image, and a process of combining the non-real image-container of the photographed image with the real image including the specific part and displaying the combined image.

[0084] This makes it possible to display a realistic image of a specific part of a person in an image while respecting the privacy of the person in the image.

[0085] In the image display device, the storage device may further store attribute information about a viewer of the real image and display conditions about at least one of the specific portion and the subject, and the processor may display the real image including the specific portion or the specific portion of the specific subject that can be displayed to the viewer based on the attribute information and the display conditions.

[0086] This makes it possible to control the display target by a combination of the viewer's attributes and the subject's attributes, or by a combination of the viewer's attributes and the type of specific body part. For example, it is possible to display a real image of "Worker A" to a viewer with a certain authority, and also to display a "hands-on image" to a viewer with a certain authority, regardless of the worker. Ultimately, it becomes possible to display a real image of a specific body part of a person in the image while giving more consideration to the privacy of the person in the image.

[0087] In the image display device, the storage device may further store information regarding a specific position in the space where the real image including the specific part should be displayed. The processor may perform a process of identifying a location in the space of a viewer of the real image, and when the location reaches or approaches within a certain distance from the specific position, display the real image including the specific part, which should be displayed for the specific position, at the specific position in the space.

[0088] This allows a viewer or their avatar in a real space including augmented reality or a virtual reality space to automatically view the actual image they need when they reach that location, for example, when they need to view the work of a skilled worker in that location. This in turn allows a real image of a specific part of a person to be displayed while respecting the privacy of the person in the image.

[0089] In the image display device, the processor may receive an instruction from the subject or a manager of the subject regarding the specific part, and store information about the specific part that is the target of the instruction in the storage device.

[0090] This allows, for example, a site manager or worker to specify a specific body part by operating the VR / AR goggles they are wearing, which in turn makes it possible to display a realistic image of a specific body part of a person in the image while respecting the privacy of the person in the image.

[0091] In the image display device, the storage device may further store information regarding a prohibited area that prohibits display of the actual image when the specific part approaches or overlaps within a certain distance from the prohibited area, and the processor may erase the display of the subject in the prohibited area or perform specific image control when the specific part approaches within a certain distance from the prohibited area or overlaps with the prohibited area.

[0092] This makes it possible to avoid a situation where, for example, when a worker's hands, i.e., a specific body part, moves to the worker's face, an actual image of the face is also displayed on the viewer's device when trying to display an actual image of the hands. Ultimately, it becomes possible to display a real image of a specific body part of a person in the image while giving more consideration to the privacy of the person in the image.

[0093] In addition, in the image display device, the processor may perform a recognition process on a code assigned to at least one of the specific part or a hidden part to be hidden in the subject in the captured image, and extract an actual image including the specific part based on the result of the recognition process.

[0094] This allows for efficient identification of specific or hidden body parts based on the results of reading a code (e.g., a QR code (registered trademark)) attached to the target body part. This in turn allows for the display of a realistic image of a specific body part of a person in an image while giving greater consideration to the privacy of the person in the image.

[0095] In addition, in the image display device, when the display area of ​​the specific part in the captured image falls below a standard, the processor may extract an actual image including the specific part from another captured image in which the subject is captured from a different direction than the captured image, and display the actual image in the captured image.

[0096] This allows the viewer to view an image of a blind spot, and ultimately allows a realistic image of a specific part of a person in the image to be displayed while giving more consideration to the privacy of the person.

[0097] In the image display device, the storage device may further store information on whether each subject can be displayed in a space where the subject is displayed, and the processor may perform display control of the subject including the specific body part or an avatar of the subject for each subject in the space based on the information on whether the subject can be displayed.

[0098] This makes it possible to display a real image of a subject (or a person corresponding to the subject) in an augmented reality or virtual reality space, or to make the subject appear as a non-real image, depending on whether or not the subject (or a person corresponding to the subject) has consented to disclosure. This in turn makes it possible to display a real image of a specific part of a person in an image, while giving more consideration to the privacy of the person in the image.

[0099] In the image display device, the storage device may further store attribute information of the subjects. The processor may identify a subject having a specific attribute based on the attribute information, and display an actual image of the identified subject including the specific part around an actual image of another subject not having the attribute including the specific part.

[0100] This makes it possible to display, for example, a real image of a veteran worker's specific body part on the goggles (terminal) of another worker, thereby enabling the display of a real image of the specific body part of the person in the image while giving more consideration to the privacy of the person in the image.

[0101] In addition, in the image display device, the processor may identify a group of subjects located in a predetermined space based on the positions of the subjects, and display real images of each of the subjects in the group of subjects on a terminal of a person corresponding to each of the subjects in the group of subjects. When a viewer outside the space views a captured image of the group of subjects in the space, the processor may perform the non-real image generation process for each of the subjects in the group of subjects, and display the non-real image on the terminal of the viewer.

[0102] This allows different controls for non-realistic imaging (masking, etc.) between users who are at the same site and users who are remotely accessing from a different location, rather than at the site. This in turn allows for the display of realistic images of specific parts of a person in the image while giving more consideration to the privacy of the person.

[0103] In addition, in the image display device, the processor may perform display control to suggest a non-real image area designated by an administrator when displaying a real image of a subject present in a specified space on a terminal of a person present in the space.

[0104] This makes it possible to display a real image of the non-real image area designated by the administrator on the goggles (terminal) of the target worker, while superimposing a guide or the like that defines the real image area. This in turn makes it possible to display a real image of a specific part of a person in the image, while giving more consideration to the privacy of the person in the image. [Explanation of symbols]

[0105] 10: Image display system, 100: Image display device, 200: Management terminal, 300: User terminal, 400: Photography system

Claims

1. a storage device for storing captured images including actual images of the subject; a processor that executes a process of extracting a real image including a specific part of the subject from the real images, a process of converting a region other than the specific part in the photographed image into a non-real image, and a process of combining the real image including the specific part with the photographed image that has been converted into a non-real image and displaying the combined image; An image display device comprising:

2. The storage device includes: attribute information relating to a viewer of the actual image and a display condition relating to at least one of the specific portion and the subject; The processor: displaying an actual image including a specific portion or a specific portion of a specific subject that can be displayed to the viewer based on the attribute information and the display conditions; 2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

3. The storage device includes: further storing information about a specific position in space where a real image including the specific part is to be displayed; The processor: a process of identifying a location of a viewer of the real image in the space, and when the location reaches the specific position or approaches within a certain distance, displaying the real image including the specific part to be displayed for the specific position at the specific position in the space; 2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

4. The processor: receiving an instruction from the subject or a manager of the subject regarding the specific part, and storing information on the specific part that is the target of the instruction in the storage device; 2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

5. The storage device includes: further storing information regarding a prohibited area in which display of the actual image is prohibited when the specific portion of each area of ​​the subject approaches or overlaps within a certain distance; The processor: When the specific part approaches within a certain distance from the prohibited area or overlaps with the prohibited area, the display of the subject in the prohibited area is erased or specific image control is performed.

2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

6. The processor: a recognition process is performed on a code assigned to at least one of the specific portion and a concealed portion to be hidden in the subject in the photographed image, and an actual image including the specific portion is extracted based on a result of the recognition process; 2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

7. The processor: When the display area of ​​the specific part in the photographed image is equal to or smaller than a reference value, an actual image including the specific part is extracted from another photographed image obtained by photographing the subject from a different direction from that of the photographed image, and is displayed in the photographed image.

2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

8. The storage device includes: further storing information on whether each subject can be displayed in the space where the subject is displayed; The processor: performing display control of the subject including the specific part or an avatar of the subject for each subject in the space based on the information on whether or not the subject can be displayed; 2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

9. The storage device includes: It also retains the subject's attribute information, The processor: a subject having a specific attribute is identified from among the subjects based on the attribute information, and an actual image including the specific part of the subject is displayed around actual images including the specific part of other subjects not having the attribute; 2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

10. The processor: Identifying a group of subjects located in a predetermined space based on the positions of the subjects, and displaying actual images of each of the subjects in the group of subjects on a terminal of a person corresponding to each of the subjects in the group of subjects; When a viewer outside the space views a photographed image relating to the group of subjects in the space, the non-real image processing is performed on each of the subjects in the group of subjects, and the non-real image processing is displayed on a terminal of the viewer.

2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

11. The processor: When displaying a real image of a subject present in a predetermined space on a terminal of a person present in the space, a display control is performed to suggest a non-real image area designated by an administrator.

2. The image display device according to claim 1, wherein the image display device is a display device having a plurality of display areas.

12. The image display device A captured image including an actual image of the subject is stored in a storage device; A process of extracting a real image including a specific part of the subject from the real image, a process of converting an area of ​​the photographed image other than the specific part into a non-real image, and a process of combining the real image including the specific part with the photographed image that has been converted into a non-real image and displaying the combined image. An image display method comprising:

Images