Video providing system and user terminal
The video providing system uses person re-identification and image processing to accurately track and conceal specific individuals in videos, addressing privacy concerns by ensuring precise concealment without affecting the display of intended persons.
Patent Information
- Application Number
- JP2023075607
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-01
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2043-05-01
AI Technical Summary
Existing video provision systems fail to appropriately protect the privacy of individuals appearing in the video, particularly when specific persons need to be concealed while others are displayed.
A video providing system that utilizes person re-identification processing to track and conceal specific individuals in videos based on their position information, performing image processing to mask or anonymize them without affecting the display of intended persons, using a combination of cameras and machine learning techniques for accurate identification and concealment.
The system effectively protects the privacy of individuals by accurately tracking and concealing specific persons in videos, preventing misidentification and ensuring high-precision image processing to maintain the privacy of those to be concealed while displaying the intended individuals.
Smart Images

Figure 2024160515000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a technique for providing video to a user. [Background technology]
[0002] Patent Document 1 discloses a video surveillance system. A surveillance camera is installed with an angle of view that overlooks a personal authentication device installed near a security gate. The video surveillance system captures video of the area near the security gate. The video surveillance system extracts people from the video captured by the surveillance camera, and identifies the person who has been authenticated for access based on the position and behavior of the extracted person. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2010-154134 A Summary of the Invention [Problem to be solved by the invention]
[0004] When providing video to users, it is desirable to appropriately protect the privacy of people appearing in the video.
[0005] One objective of the present disclosure is to provide a technology that can appropriately protect the privacy of people appearing in a video when the video is provided to a user. [Means for solving the problem]
[0006] One aspect of the present disclosure relates to an image providing system that provides an image to a user. The video providing system includes one or more processors. A concealed person is a person other than the displayed person that the user wishes to see. The one or more processors Acquire one or more images captured by one or more cameras; By performing a person re-identification process, each person appearing in one or more images is tracked to obtain person information including position information of each person in the one or more images; performing image processing on one or more images based on the person information so as to conceal the person to be concealed without concealing the person to be displayed; Displaying the image or images after image processing on the user's user terminal. It is configured as follows. Effect of the Invention
[0007] According to the present disclosure, when a video is provided to a user, a person to be anonymized that appears in the video is anonymized. This makes it possible to appropriately protect the privacy of the person to be anonymized. Furthermore, by using a person re-identification process, it becomes possible to stably and accurately track each person that appears in one or more videos. This makes it possible to suppress erroneous recognition of a person to be displayed and a person to be anonymized. Therefore, it becomes possible to perform image processing with high accuracy to anonymize a person to be anonymized without anonymizing a person to be displayed. As a result, it becomes possible to more appropriately protect the privacy of a person to be anonymized. [Brief description of the drawings]
[0008] [Figure 1] 1 is a conceptual diagram for explaining an overview of a video providing system according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of a configuration of a video providing system according to an embodiment; [Diagram 3] 1 is a block diagram showing an example of a functional configuration of a person information management system according to an embodiment; [Figure 4] FIG. 4 is a conceptual diagram illustrating an example of person information according to the embodiment. [Diagram 5] 1 is a conceptual diagram for explaining an example of processing performed by a video providing system according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] 1. Overview FIG. 1 is a conceptual diagram for explaining an overview of a video providing system 10 according to the present embodiment. The video providing system 10 acquires a video VID captured by a camera CAM installed in a predetermined area. The video providing system 10 then provides (presents) the video VID to a user. More specifically, the video providing system 10 includes a user terminal, and displays the video VID on the user terminal. For example, in a monitoring service that allows parents to monitor their children, a video VID showing the child is displayed in real time on the parent's user terminal. The parent can monitor the child by watching the video VID displayed on the user terminal.
[0010] For example, person P11 is an object that user A wants to watch. The person P11 is captured in a video VID-1 captured by a camera CAM-1. However, other persons P12 and P13 are also captured in the video VID-1. In this case, when providing (presenting) the video VID-1 to user A, it is desirable to protect the privacy of the persons P12 and P13 other than the person P11. Therefore, the video providing system 10 performs image processing on the video VID-1 so as to anonymize the persons P12 and P13. In other words, the video providing system 10 performs image processing on the video VID-1 so that the persons P12 and P13 are not identified. For example, image processing is performed so that at least the faces of the persons P12 and P13 are masked. Then, the video providing system 10 displays the video VID-1' after image processing on the user terminal of user A.
[0011] As another example, person P13 is a target that user B wants to watch. In this case, the video providing system 10 performs image processing on the video VID-1 so as to conceal persons P11 and P12 other than person P13. Then, the video providing system 10 displays the image-processed video VID-1' on the user terminal of user B.
[0012] Also, persons P21 and P22 appear in the video VID-2 captured by the camera CAM-2. Here, it is assumed that person P13 appearing in the video VID-1 and person P21 appearing in the video VID-2 are the same. If the angles of view of the cameras CAM-1 and CAM-2 do not overlap, the timing at which person P13 appears in the video VID-1 and the timing at which person P21 appears in the video VID-2 are different. If the angles of view of the cameras CAM-1 and CAM-2 partially overlap, the timing at which person P13 appears in the video VID-1 and the timing at which person P21 appears in the video VID-2 may be the same. In either case, the video providing system 10 correctly recognizes (determines) that person P13 appearing in the video VID-1 and person P21 appearing in the video VID-2 are the same, using a method described below. Then, the video providing system 10 performs image processing on the video VID-2 so as to conceal person P22 other than person P21. Then, the video providing system 10 displays the image-processed video VID-2' on the user terminal of user B. For example, the video providing system 10 switches between displaying the video VID-1' and the video VID-2'.
[0013] As yet another example, user C monitors the background (e.g., traffic flow) other than the people in video VID-1. In this case, video providing system 10 performs image processing on video VID-1 so as to conceal all of people P11, P12, and P13. Then, video providing system 10 displays video VID-1' after image processing on user C's user terminal.
[0014] More generally, a person that a user wants to see is hereinafter referred to as a "display target person TD". The display target person TD is different for each user. Typically, the display target person TD is specified by the user. There may be no display target person TD. On the other hand, a person who is concealed so that he / she cannot be identified by the user is hereinafter referred to as a "concealment target person TX". The concealment target person TX is a person other than the display target person TD. For example, for user A, person P11 is a display target person TD, and persons P12 and P13 are concealment target persons TX. As another example, for user B, persons P13 and P21 are display target persons TD, and persons P11, P12, and P22 are concealment target persons TX. For user C, persons P11, P12, and P13 are concealment target persons TX.
[0015] The video providing system 10 acquires one or more videos VID-1 to VID-N captured by one or more cameras CAM-1 to CAM-N. Here, N is an integer equal to or greater than 1. The video providing system 10 tracks each person appearing in the one or more videos VID-1 to VID-N and acquires in-image position information of each person appearing in the one or more videos VID-1 to VID-N. Furthermore, the video providing system 10 performs image processing on the one or more videos VID-1 to VID-N based on the in-image position information so as to conceal the anonymization target person TX without concealing the display target person TD. In other words, the video providing system 10 performs image processing so that the display target person TD is identified and the anonymization target person TX is not identified. Then, the video providing system 10 displays the one or more videos VID-1' to VID-N' after the image processing on the user terminal.
[0016] By such image processing, when providing a video VID to a user, it is possible to appropriately protect the privacy of the person TX to be concealed who appears in the video VID. Even if the person TX to be concealed is different for each user, it is possible to appropriately protect the privacy of the person TX to be concealed. In other words, appropriate privacy control is possible for each user.
[0017] In order to ensure the accuracy of image processing for concealing the anonymization target person TX without concealing the display target person TD, it is necessary to accurately track the display target person TD and the anonymization target person TX. If the anonymization target person TX is erroneously recognized as the display target person TD, or if the display target person TD is erroneously recognized as the anonymization target person TX, the accuracy of the image processing decreases. The decrease in the accuracy of the image processing leads to a decrease in the accuracy of privacy protection.
[0018] Therefore, according to this embodiment, "Person Re-identification" is used to track each person appearing in one or more videos VID-1 to VID-N. Person re-identification is a technology for identifying the same person from among the images of one or more videos VID-1 to VID-N. More specifically, a person is detected from the images constituting the video VID, the overall feature amount of the detected person is extracted, and a person re-identification process is performed based on the extracted feature amount. The feature amount is extracted based on a person re-identification model based on machine learning.
[0019] Such person re-identification processing makes it possible to stably and accurately track each person appearing in one or more videos VID-1 to VID-N. For example, in FIG. 1, person P13 appearing in video VID-1 and person P21 appearing in video VID-2 are the same, but this person can also be recognized with high accuracy by the person re-identification processing. That is, the person re-identification processing suppresses erroneous recognition of the display target person TD and the concealment target person TX. Therefore, it becomes possible to perform image processing with high accuracy to conceal the concealment target person TX without concealing the display target person TD. As a result, it becomes possible to more appropriately protect the privacy of the concealment target person TX.
[0020] A specific example of the video providing system 10 according to the present embodiment will be described in detail below.
[0021] 2.Example of a video provision system 2 is a block diagram showing an example of the configuration of a video providing system 10 according to this embodiment. The video providing system 10 includes a video management system 100, a person information management system 200, and a user terminal 300 used by a user.
[0022] 2-1.Video management system The video management system 100 acquires and manages one or more videos VID-1 to VID-N captured by one or more cameras CAM-1 to CAM-N. The video database 130 is a database of one or more videos VID-1 to VID-N acquired during a predetermined period.
[0023] The video management system 100 includes one or more processors 110 (hereinafter simply referred to as "processor 110") and one or more storage devices 120 (hereinafter simply referred to as "storage device 120"). The processor 110 executes various processes. For example, the processor 110 includes a CPU. The storage device 120 stores various information required for the processes. The video database 130 is also stored in the storage device 120. Examples of the storage device 120 include HDD, SSD, volatile memory, non-volatile memory, etc. The functions of the video management system 100 may be realized by cooperation between the processor 110, which executes a computer program, and the storage device 120.
[0024] 2-2. Person information management system The person information management system 200 acquires one or more videos VID-1 to VID-N captured by one or more cameras CAM-1 to CAM-N. The person information management system 200 then acquires and manages person information PSN regarding each person appearing in the one or more videos VID-1 to VID-N. For example, the person information management system 200 performs a person re-identification process to track each person appearing in the one or more videos VID-1 to VID-N with high accuracy. The person information PSN includes in-image position information indicating the position of each person in the one or more videos VID-1 to VID-N. The person database 230 is a database of person information PSN acquired during a predetermined period.
[0025] The person information management system 200 includes one or more processors 210 (hereinafter simply referred to as "processor 210") and one or more storage devices 220 (hereinafter simply referred to as "storage device 220"). The processor 210 executes various processes. For example, the processor 210 includes a CPU. The storage device 220 stores various information required for the processes. The person database 230 is also stored in the storage device 220. Examples of the storage device 220 include HDD, SSD, volatile memory, non-volatile memory, etc. The functions of the person information management system 200 may be realized by cooperation between the processor 210 that executes a computer program and the storage device 220.
[0026] 3 is a block diagram showing an example of a functional configuration of the person information management system 200. The person information management system 200 includes a person detection unit 211, a tracking unit 212, a face authentication unit 213, a posture estimation unit 214, and a person information provision unit 215 as functional blocks.
[0027] One or more videos VID-1 to VID-N are input to the person detection unit 211. Each video VID includes a series of images (frames). The person detection unit 211 performs a person detection process to detect a person in each image. A bounding box represents the position of a person detected in an image. The person detection unit 211 acquires information on the bounding box of a person in each image. Note that the person detection process is a well-known technique, and the method is not particularly limited. For example, YOLOX is used as the person detection unit 200.
[0028] The tracking unit 212 automatically tracks each person appearing in one or more videos VID-1 to VID-N.
[0029] For example, the tracking unit 212 tracks the same person in a video VID captured by a certain camera CAM. Specifically, the tracking unit 212 tracks bounding boxes representing the same person in a series of images constituting the video VID. Multiple bounding boxes representing the same person in multiple time steps are associated with each other. A "track" is a set of multiple bounding boxes representing the same person in multiple time steps. Such tracking processing is a well-known technique, and the method is not particularly limited. For example, ByteTrack is used.
[0030] Moreover, the tracking unit 212 tracks the same person appearing in one or more videos VID-1 to VID-N by performing a person re-identification process. In particular, the tracking unit 212 tracks the same person appearing in different videos VID captured by different cameras CAM by performing a person re-identification process. More specifically, the tracking unit 212 extracts a feature amount of a person for re-identification process (hereinafter, referred to as "ReID feature amount") based on an image of a part of the person. A partial image surrounded by each bounding box in an image corresponds to an image of the person. The tracking unit 212 extracts the ReID feature amount from the partial image by using a ReID model based on machine learning. In particular, the tracking unit 212 extracts the ReID feature amount for a plurality of bounding boxes constituting a track. The extracted plurality of ReID feature amounts are associated with the track. Then, the tracking unit 212 performs matching based on the ReID feature amount and determines whether or not two or more different tracks are the same. That is, the tracking unit 212 judges whether two or more people appearing in one or more videos VID-1 to VID-N are the same or not based on the ReID feature. Note that such person re-identification processing is a well-known technique, and the method is not particularly limited. The ReID model may be a model based on a Transformer.
[0031] The face authentication unit 213 holds face images of people registered in advance. The face authentication unit 213 performs face authentication processing based on the face images registered in advance to individually identify a person appearing in one or more videos VID-1 to VID-N. For example, a face image of a display target person TD is registered in advance by a user. The face authentication unit 213 performs face authentication processing based on the face images registered in advance to individually identify a display target person TD appearing in one or more videos VID-1 to VID-N. The individually identified person is associated with the above-mentioned track. Note that such face authentication processing is a well-known technique, and the method is not particularly limited.
[0032] The pose estimation unit 214 performs a pose estimation process to estimate the pose of a person based on an image of a part of the person. A partial image surrounded by each bounding box in an image corresponds to an image of the person. The pose estimation unit 214 extracts key points from the partial image by using a pose estimation model based on machine learning, and estimates the pose of the person. In particular, the pose estimation unit 214 estimates the pose of the person in each bounding box that constitutes a track. The estimated pose information is associated with the track. Note that the pose estimation process is a well-known technique, and the method is not particularly limited. For example, TransPose is used as the pose estimation unit 214.
[0033] The tracking unit 212 generates person information PSN on each person appearing in one or more videos VID-1 to VID-N. For example, the person information PSN is generated for each track.
[0034] FIG. 4 shows an example of person information PSN. In the example shown in FIG. 4, the person information PSN shows the correspondence relationship between track ID, person ID, camera ID, timestamp, in-image position information, posture information, and the like. The track ID is identification information of the track. The person ID is identification information of a person associated with the track and identified by the face authentication unit 213. Information of each image including each bounding box constituting the track is obtained from one or more videos VID-1 to VID-N. The camera ID is identification information of the camera CAM that captured each image. The timestamp is the time when each image was captured. The in-image position information is position information of a person (bounding box) in each image, and is obtained by the tracking unit 212 tracking the person. The posture information is information of the posture of the person in each image, and is obtained by the posture estimation unit 214.
[0035] The person database 230 is a database of person information PSN acquired during a predetermined period of time. The tracking unit 212 generates and updates the person database 230 by accumulating the person information PSN.
[0036] The person information providing unit 215 acquires the necessary person information PSN from the person database 230 in response to a request from the user terminal 300 , and provides the person information PSN to the user terminal 300 .
[0037] 2-3. User terminal The user terminal 300 includes one or more processors 310 (hereinafter simply referred to as "processor 310"), one or more storage devices 320 (hereinafter simply referred to as "storage device 320"), and a user interface 330. The processor 310 executes various processes. For example, the processor 310 includes a CPU. The storage device 320 stores various information required for the processes. Examples of the storage device 320 include HDD, SSD, volatile memory, non-volatile memory, etc. The functions of the user terminal 300 may be realized by cooperation between the processor 310 that executes a computer program and the storage device 320.
[0038] The user interface 330 is an interface that receives information from a user and outputs information to the user. The user interface 330 includes an input device and an output device. Examples of the input device include a touch panel, a keyboard, and a button. Examples of the output device include a display device and a speaker.
[0039] The user terminal 300 is capable of communicating with the video management system 100 and the person information management system 200. The user terminal 300 acquires the necessary video VID from the video management system 100. The user terminal 300 also acquires the necessary person information PSN from the person information management system 200. The video VID and the person information PSN are stored in the storage device 320. The user terminal 300 performs image processing on the video VID based on the person information PSN. The user terminal 300 then displays the video VID' after the image processing on a display device, that is, provides (presents) the video VID' after the image processing to the user. An example of processing by the user terminal 300 will be described in more detail below.
[0040] 3. Example of processing by user terminal 3-1. First Example The user specifies the person ID of the person TD to be displayed using the user interface 330 (input device). The user terminal 300 transmits a request including the person ID of the specified person TD to be displayed to the person information management system 200.
[0041] The personal information management system 200 acquires personal information PSN related to the designated display target person TD from the person database 230. For example, the personal information PSN related to the display target person TD includes the personal information PSN of the concealment target person TX who is captured on the same camera CAM as the display target person TD. The personal information PSN of such a concealment target person TX has a common camera ID with the personal information PSN of the display target person TD. The personal information PSN related to the display target person TD may include the personal information PSN of the display target person TD itself. The personal information management system 200 transmits the personal information PSN acquired in this manner to the user terminal 300.
[0042] The user terminal 300 checks the camera ID included in the person information PSN acquired from the person information management system 200. Then, the user terminal 300 acquires the video VID captured by the camera CAM corresponding to the camera ID. For example, the user terminal 300 transmits a request including the camera ID to the video management system 100. The video management system 100 selects the video VID captured by the camera CAM corresponding to the camera ID from the video database 130, and transmits the selected video VID to the user terminal 300. As another example, the user terminal 300 may first acquire videos VID-1 to VID-N from the video management system 100, and then select from among them the video VID captured by the camera CAM corresponding to the camera ID.
[0043] The user terminal 300 performs image processing on the video VID so as to conceal the anonymization target person TX based on the person information PSN of the anonymization target person TX. More specifically, the user terminal 300 temporally associates the image and the person information PSN based on the timestamp of each image (frame) included in the video VID and the timestamp included in the person information PSN. The in-image position information of the anonymization target person TX in the image is obtained from the person information PSN. Therefore, the user terminal 300 can perform image processing so as to conceal the anonymization target person TX in the image. For example, when the person information PSN includes posture information, the user terminal 300 may specify the position of the face of the anonymization target person PSN based on the posture information, and may mask the face part of the anonymization target person PSN. As another example, the user terminal 300 may mask the entire position (bounding box) of the anonymization target person TX. It is not necessary to mask the display target person TD.
[0044] The user terminal 300 then displays the image-processed video VID' on a display device. When the video VID in which the display target person TD appears is switched, the user terminal 300 switches the video VID' to be displayed in conjunction with the video VID.
[0045] 3-2. Second Example The user specifies a camera using the user interface 330 (input device). The user terminal 300 transmits a request including the camera ID of the specified camera to the person information management system 200. The person information management system 200 acquires person information PSN including the specified camera ID from the person database 230, and transmits the acquired person information PSN to the user terminal 300. In addition, the user terminal 300 acquires video VID captured by the camera CAM corresponding to the camera ID, similar to the first example above.
[0046] In the second example, the person to be displayed TD is not specified, so all people appearing in the video VID are the person to be concealed TX. As in the first example, the user terminal 300 performs image processing on the video VID so as to conceal the person to be concealed TX. The user terminal 300 then displays the image VID' after the image processing on the display device.
[0047] 3-3.Third Example A combination of the first and second examples above is also possible. In that case, the user specifies the camera and the person to be displayed TD.
[0048] 3-4. Fourth Example As described above, the personal information management system 200 acquires the personal information PSN by dropping various processes. Therefore, the sampling rate (acquisition rate) of the personal information PSN may be lower than the frame rate of the video VID. For example, the frame rate of the video VID may be 30 FPS, while the sampling rate of the personal information PSN may be 10 FPS.
[0049] For example, in Fig. 5, the video VID includes frames FR1 to FR10. The time stamps of the frames FR1 to FR10 are ts1 to ts10, respectively. On the other hand, the person information PSN is only available for time stamps ts1, ts4, ts7, and ts10. Even in such a case, it is desirable to accurately perform the concealment process based on the person information PSN.
[0050] Therefore, the user terminal 300 selects frames FR1, FR4, FR7, and FR10 corresponding to the timestamp of the person information PSN from among the frames FR1 to FR10 constituting the video VID. Then, the user terminal 300 selectively performs image processing on the selected frames FR1, FR4, FR7, and FR10. After that, the user terminal 300 selectively displays the frames FR1, FR4, FR7, and FR10 after image processing on a display device. The remaining frames are not displayed. Although the frame rate of the displayed video VID' decreases, the anonymization of the person to be anonymized TX is performed with high accuracy. In other words, the privacy of the person to be anonymized TX is appropriately protected.
[0051] 4. Variations A part of the processing by the user terminal 300 may be executed by the person information management system 200. For example, the person information management system 200 may perform image processing for each user, generate a video VID' after the image processing for each user, and transmit the video VID' after the image processing to the user terminal 300 of the user. This also provides the same effect as the above-mentioned embodiment. [Explanation of symbols]
[0052] 10...Video providing system, 100...Video management system, 200...Personal information management system, 300...User terminal, PSN...Personal information, TD...Person to be displayed, TX...Person to be concealed, VID...Video
Claims
1. A video providing system for providing a video to a user, comprising: One or more processors; The concealment target person is a person other than the display target person that the user wishes to see, The one or more processors: Acquire one or more images captured by one or more cameras; By performing a person re-identification process, each person appearing in the one or more images is tracked, and person information including position information of each person in the one or more images is acquired; performing image processing on the one or more images based on the person information so as to conceal the person to be concealed without concealing the person to be displayed; Displaying the one or more images after the image processing on a user terminal of the user. It was configured as Video delivery system.
2. The video providing system according to claim 1 , When a sampling rate of the person information is lower than a frame rate of the one or more videos, the one or more processors select a frame corresponding to a time stamp of the person information from among frames constituting the one or more videos, perform the image processing on the selected frame, and selectively display the selected frame after the image processing on the user terminal. Video delivery system.
3. 3. The video providing system according to claim 1, The one or more processors further estimate a pose of each of the persons in the one or more images; the person information includes the position information and the posture information of each person, In the image processing, the one or more processors at least mask a face of the person to be concealed based on the person information. Video delivery system.
4. 3. The video providing system according to claim 1, The one or more processors further perform a face recognition process to identify the display target person appearing in the one or more images. Video delivery system.
5. A user terminal used by a user, a processor for acquiring one or more images captured by one or more cameras; The person information management system performs a person re-identification process to track each person appearing in the one or more videos and acquire person information including position information of each person in the one or more videos; The concealed person is a person other than the displayed person that the user wishes to see, The processor, Acquire the personal information from the personal information management system; performing image processing on the one or more images based on the person information so as to conceal the person to be concealed without concealing the person to be displayed; Displaying the one or more images after the image processing. It was configured as User terminal.
Citation Information
Patent Citations
Video processing method, device, electronic equipment and readable storage medium
CN110363172A
Video coding method and device and electronic equipment
CN111614973A
Child nursing method and child nursing system and care service system
JP2002269209A
Image encryption system and method
JP2005229265A
Method and apparatus for automatically blurring faces
JP2005512203A
