Management system for camera images

Abstract

The invention relates to a camera image management system, which can protect the privacy of a person reflected on a camera image and reduce the size of data required for playing a scene reflected on the camera image. The processing circuit generates identification information of an object reflected on the camera image by performing object identification processing on the camera image. In addition, the processing circuit generates language information of a scene reflected on the camera image through language processing of the camera image. In addition, the processing circuit generates scene information in which the identification information of the person and the language information of the scene are associated, and stores the scene information in the storage device. The processing circuit also performs reproduction processing of the scene reflected on the camera image based on the scene information. In the reproduction process, an abstracted image of the space of the camera image is rendered, and an abstracted image of the person is rendered on the abstracted image of the space. In the reproduction processing, title information generated according to language information is added to the abstraction image of the space to generate a reproduced image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a system that manages camera footage. BACKGROUND

[0002] Japanese Patent Application Publication No. 2006-295251 discloses a system that performs image processing on monitoring camera footage and outputs the footage from an external monitor. The existing system extracts a person from the monitoring camera footage, detects a posture and a behavior of the extracted person, and calculates an importance degree that the detected posture and behavior have. In addition, the existing system determines a range of the monitoring camera footage that becomes an object of image processing based on the calculated importance degree. In the image processing, privacy protection processing such as mosaic processing, smoothing processing, or the like is performed on the range that becomes the object of the image processing. That is, in the existing system, footage on which image processing corresponding to the importance degree that the posture and the behavior of the person have is performed is output from the external monitor.

[0003] The existing system also performs analysis of a scene of the monitoring camera footage. Furthermore, when it is determined in the analysis that the footage scene is abnormal, the footage on which the image processing is performed is attached with text information indicating that it is an abnormal scene.

[0004] As documents that indicate the technical level of the technical field related to the present disclosure, in addition to Japanese Patent Application Publication No. 2006-295251, Japanese Patent Application Publication No. 2022-056533, Japanese Patent Application Publication No. 2019-144830, and Japanese Patent Application Publication No. 2002-024962 can be exemplified.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2006-295251

[0006] Patent Document 2: Japanese Patent Application Publication No. 2022-056533

[0007] Patent Document 3: Japanese Patent Application Publication No. 2019-144830

[0008] Patent Document 4: Japanese Patent Application Publication No. 2002-024962

[0009] In the existing system, the calculation of the importance degree that the posture and the behavior of the person have is performed by referring to a score table that is created in advance. However, there is a possibility that the range that becomes the object of the image processing cannot be accurately determined depending on the granularity of the information of the score table, and the privacy protection of the person appearing in the monitoring camera footage becomes insufficient.

[0010] According to the existing system, by attaching text information indicating an abnormal scene to the output image from the external monitor, it is expected that the image scene is easily transmitted to the viewer of the external monitor. However, in order to play such an output image with text information, it is necessary to combine and save the image subjected to image processing with the text information, and there is a problem that the data size of the image becomes large. SUMMARY

[0011] An object of the present disclosure is to provide a technology capable of reducing the size of data required for playing a scene represented in a camera image while protecting the privacy of a person represented in the camera image.

[0012] The present disclosure relates to a management system of a camera image, having the following features.

[0013] The above system has a storage device, a processing circuit, and a display device. The above camera image is stored in the above storage device. The above processing circuit is configured to perform various processes. The above display device is configured to output an image.

[0014] The above processing circuit is configured to generate identification information of an object represented in the above camera image by object recognition processing of the above camera image stored in the above storage device, generate language information of a scene represented in the above camera image by verbalization processing of the above camera image stored in the above storage device, generate scene information and store it in the above storage device in the case where the identification information of the object includes identification information of a person, perform reproduction processing of a scene represented in the above camera image based on the above scene information stored in the above storage device, the scene information being information in which the identification information of the person and the language information of the scene generated by the above verbalization processing of the above camera image in which the person is recognized are associated.

[0015] The above reproduction processing includes rendering an abstract image of a space represented in the above camera image based on the identification information of a static object included in the above scene information, rendering an abstract image of the person represented in the above camera image on the abstract image of the space based on the identification information of the person included in the above scene information, generating title information of the scene based on the language information of the scene included in the above scene information, and adding the title information to the abstract image of the space in which the abstract image of the person is rendered, generating a reproduction image for output from the above display device.

[0016] According to this disclosure, a scene representation process of a scene in a camera image is performed by adding a title image to an image that renders an abstract image of a person in the camera image on an abstract image of the space projected onto the camera image, along with linguistic information of the scene projected onto the camera image. Here, the abstract images of the space and the person are rendered based on object recognition information generated through object recognition processing of the camera image. Therefore, compared to using raw image data or image data with privacy protection processing implemented in the prior art, the data size required for playing the image scene can be reduced. Furthermore, since an abstract image of the person is used, the privacy of that person can also be protected. Attached Figure Description

[0017] Figure 1 This is a block diagram illustrating an example of the overall configuration of a camera image management system according to an embodiment of the present disclosure.

[0018] Figure 2 It means Figure 1 A block diagram illustrating an example of the functional configuration of a data processing device.

[0019] Figure 3 Yes Figure 2 The diagram illustrates an example of the reproduction process performed by the reproduction unit shown.

[0020] Figure 4 Yes Figure 2 The diagram illustrates other examples of the reproduction processing performed by the reproduction unit shown.

[0021] Figure 5 It means Figure 1 Block diagrams illustrating other examples of the functional configuration of the data processing apparatus shown. Detailed Implementation

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the drawings, the same or equivalent parts are labeled with the same reference numerals and their descriptions are simplified or omitted.

[0023] 1. Example of overall composition

[0024] Figure 1 This is a block diagram illustrating an example of the overall configuration of a camera image management system according to an embodiment of the present disclosure. Figure 1 The document describes the configuration of a management system, including a camera 10, a data processing device 20, a display device 30, and an input device 40, as described in the embodiment. The camera 10, display device 30, and input device 40 communicate with the data processing device 20 via a communication network (not shown). The communication network is not particularly limited and can be either wired or wireless.

[0025] The cameras 10 are disposed in any space in a house and outside the house. The disposition positions and the photographing ranges of the cameras 10 are known. The cameras 10 acquire images of their photographing ranges. The cameras 10 transmit the images acquired by the cameras 10 (i.e., camera images VD) to the data processing apparatus 20 together with their own identification information. The total number of the cameras 10 is at least one. A part or all of the photographing ranges of two or more cameras can overlap.

[0026] The data processing apparatus 20 is provided with at least one processing circuit 21 and at least one storage apparatus 22. As the processing circuit 21, a general-purpose processor, a special-purpose processor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and the like can be exemplified. As the storage apparatus 22, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a volatile memory, a non-volatile memory, and the like can be exemplified.

[0027] The processing circuit 21 develops various programs stored in the storage apparatus 22 and performs processing of various data stored in the storage apparatus 22. The data processing performed by the processing circuit 21 includes processing of the camera images VD. In the data processing of the camera images VD, a two-dimensional or three-dimensional virtual image is generated from an image constituting the camera images VD and is output to the display apparatus 30. Hereinafter, for convenience of explanation, an image constituted by a virtual image will also be referred to as “image VD VR”. In addition, the camera image VD, which is a generation source of the virtual image, will also be referred to as “image VD OR”. Details of an example of the data processing of the image VD OR will be described later.

[0028] The display apparatus 30 displays various data. The various data displayed on the display apparatus 30 is provided to a user of the management system. The various data displayed on the display apparatus 30 includes the image VD VR. As the display apparatus 30, a liquid crystal display, an organic EL display, a head-up display (Head-Up Display), and the like can be exemplified.

[0029] Input device 40 is managed by the system's user operations. Examples of input device 40 include keyboard, mouse, touch panel, microphone, etc. As input information particularly relevant to the implementation method, retrieval information SAR can be cited. Retrieval information SAR is information used to retrieve scenes displayed in the image VD_OR. Retrieval information SAR includes string information. When input device 40 is a microphone, the audio information input from the microphone is converted into string information.

[0030] 2. Example of functional composition

[0031] Figure 2 It means Figure 1 A block diagram illustrating an example of the functional configuration of the data processing device 20 shown. Figure 2 The text describes the object recognition unit 23, speech recognition unit 24, scene information generation unit 25, image recording unit 26, scene information recording unit 27, and playback unit 28 as functional modules of the data processing device 20. These functional modules, for example, are... Figure 1 The processing circuit 21 shown is implemented in conjunction with the storage device 22.

[0032] The object recognition unit 23 performs "object recognition processing" to identify objects included in the image constituting the image VD_OR (hereinafter also referred to as "image IMG_OR"). In object recognition processing, objects are detected using, for example, the YOLO (You Only LookOnce) network or the SSD (Single Shot multibox Detector) network. The objects detected in object recognition processing are static objects SO and dynamic objects MO. Examples of static objects SO include buildings, structures, and natural objects. Examples of dynamic objects MO include people, robots, bicycles, and cars.

[0033] In object recognition processing, recognition information REC_SO for static objects SO and recognition information REC_MO for dynamic objects MO are generated. Recognition information REC_MO includes recognition information REC_HM for human figures HM and recognition information REC_NHM for dynamic objects other than human figures HM (robots, bicycles, cars, etc.) NHM. Recognition information REC_SO and REC_MO include information about the object type and the object's detection time. Recognition information REC_SO and REC_MO are sent to the scene information generation unit 25.

[0034] In the object recognition processing, a two-dimensional pose (2D Pose) and a three-dimensional pose (3D Pose) of the person HM are also inferred. The two-dimensional pose and the three-dimensional pose are expressed using joints, a head, hands, feet, and the like, and lines connecting the parts. Information of the two-dimensional pose and the three-dimensional pose is added to the recognition information REC HM of the person HM. In the object recognition processing, tracking of the person HM can be performed. In the object recognition processing, processing of recognizing the same person between two or more cameras 10 (person re-identification processing) can also be performed. Tracking information and re-identification information are also added to the recognition information REC HM of the person HM. Among them, the pose inference, the tracking, and the re-identification are publicly known technologies, and the method applicable to the present disclosure is not particularly limited.

[0035] The verbalization unit 24 performs "verbalization processing" of giving verbal information LAN to a scene appearing in the video VD OR. In the verbalization processing, a text that describes the scene appearing in the video VD OR is generated, for example, using a framework based on LLM models (Large Language Models). The text that describes the scene includes, for example, a description of the environment appearing in the video VD OR, a description of the person and the surrounding objects appearing in the video VD OR, a description of the interaction of the person with the surrounding objects, and the like. In other examples, a text that describes the scene appearing in the video VD OR is generated using VLM models (Vision Language Models).

[0036] Such a text output from the learning model is one example of the verbal information LAN. The verbal information LAN includes information of a start time and an end time of the scene described by the text. The verbal information LAN is transmitted to the scene information generation unit 25.

[0037] The scene information generation unit 25 generates scene information SCN by associating the information (recognition information REC SO and recognition information REC MO) received from the object recognition unit 23 and the information (verbal information LAN) received from the verbalization unit 24. For the generation of the scene information SCN, for example, the information is associated based on the time information of the scene included in the verbal information LAN and the information of the detection time of the object included in the recognition information REC SO or the recognition information REC MO. The scene information SCN is transmitted to the scene information recording unit 27.

[0038] The video recording unit 26 stores the video VD OR received from the camera 10 in the video storage unit 28. Figure 1The storage device 22 is shown. Because the image VD_OR stored in the storage device 22 has a large data size, the image VD_OR stored in the storage device 22 is compressed or deleted after a certain period of time. The image VD_OR stored in the storage device 22 can be referenced in the scene retrieval described later, or in the scene information update described later.

[0039] Scene Information Recording Department 27 Figure 1 The storage device 22 shown stores scene information SCN. Preferably, the storage device 22 storing scene information SCN and the storage device storing image VD_OR are different devices. Like the image VD_OR, the image scene information SCN stored in storage device 22 is compressed or deleted after a certain period of time. However, by storing image VD_OR and scene information SCN in separate storage devices 22, even if scene information SCN is accidentally deleted from storage device 22, scene information SCN can be regenerated from image VD_OR stored in other storage devices 22.

[0040] The reproduction unit 28 performs "reproduction processing" to reproduce the scene projected onto the image VD_OR based on the scene information SCN stored in the scene information recording unit 27. Figure 3 Yes Figure 2 The schematic diagram shown illustrates an example of the reproduction process involved in the reproduction unit 28. Figure 3 The left side depicts the real space RS, which is reflected in the image VD_OR and includes the scene information SCN. In this real space RS, there are static objects SO and dynamic objects MO (human HM and other dynamic objects NHM), which are detected through object recognition processing.

[0041] In the rendering process, rendering abstractly represents the virtual space VS of the real space RS projected onto the image VD_OR. The virtual space VS is expressed using the same world coordinate system (X, Y, Z) as the real space RS. Within the virtual space VS, a static object (virtual static object) SO_VR is defined that is equivalent to, but abstracted from, the static object SO existing in the real space RS. The composition of the virtual static object SO_VR (e.g., position, orientation, shape, size, etc.) is defined by the identification information REC_SO of the static object SO. Therefore, the composition of the virtual static object SO_VR is largely consistent with the composition of the static object SO.

[0042] In the reproduction processing, in addition, the person HM appearing in the video VD_OR is rendered in the virtual space VS. In the rendering of the person HM, the configuration (for example, position, posture, size, and the like) of the person (virtual person) HM_VR abstracted from the person HM is defined based on the recognition information REC_HM of the person HM. The posture of the virtual person HM_VR is defined based on the two-dimensional posture or the three-dimensional posture included in the recognition information REC_HM. Therefore, in the example shown in Fig. 10, the virtual person HM_VR is expressed in a skeletal shape. Figure 3 In the example shown in Fig. 10, the virtual person HM_VR is expressed in a skeletal shape.

[0043] In a case where a dynamic object NHM other than the person HM appears in the video VD_OR, in the reproduction processing, the dynamic object NHM is rendered in the virtual space VS at the same time as the person HM. In the rendering of the dynamic object NHM, the configuration (for example, position, orientation, shape, size, and the like) of the dynamic object (virtual dynamic object) NHM_VR abstracted from the dynamic object NHM is defined based on the recognition information REC_NHM of the dynamic object NHM.

[0044] In addition to the rendering of the virtual space VS, in the reproduction processing, the caption information CAP of the scene appearing in the video VD_OR is generated based on the language information LAN. The caption information CAP is, for example, an explanation of the person HM appearing in the video VD_OR or an abstract of the explanation in the language information LAN. In a case where a dynamic object NHM other than the person HM appears in the video VD_OR, it is an explanation of the interaction of the person HM and the dynamic object NHM appearing in the video VD_OR or an abstract of the explanation. The caption information CAP constitutes the video VD_VR together with the virtual space VS. In a case where the video VD_VR is output from the display device 30, the caption information CAP is displayed in the vicinity of the display region of the virtual space VS.

[0045] Returning to Figure 2 , the reproduction processing can be performed based on the search information SAR input from the input device 40. For example, in a case where the search information SAR includes the position and the time information, the reproduction section 28 determines the scene information SCN in cooperation with the scene information recording section 27 such that the position and the time information coincide with the search information SAR. Further, the reproduction section 28 performs the reproduction processing, generates the video VD_VR according to the determined scene information SCN, and outputs to the display device 30. In a case where the search information SAR further includes the keyword information KWD, in the reproduction processing, the explanation in the caption information CAP corresponding to the keyword information KWD can be highlighted (refer to Figure 4 ).

[0046] Figure 5 is a block diagram showing another example of the functional configuration of the data processing device 20 shown in Figure 1 Fig. 9. Figure 5In addition to the function modules (the object recognition unit 23 and the like) shown Figure 2 The scene information updating unit 29 is described in addition to the function modules (the object recognition unit 23 and the like) shown. These function modules are realized, for example, by the processing circuit 21 in cooperation with the storage device 22. Figure 1 The processing circuit 21 shown cooperates with the storage device 22.

[0047] The scene information updating unit 29 performs "update processing" of the update scene information SCN. In the update processing, first, a specific language is detected from the language information LAN included in the scene information SCN by cooperation with the scene information recording unit 27. The specific language is set in advance, for example, by a user of the management system. As the specific language, a language that represents a state considered to be abnormal from the viewpoint of health maintenance or traffic safety of the person HM can be exemplified. As the state considered to be abnormal from the viewpoint of health maintenance or traffic safety, a fall, a crouching, bleeding, an abnormal body posture of the person HM can be exemplified.

[0048] In a case where the specific language is detected, in the update processing, the image VD OR of the generation source of the language information LAN is determined by cooperation with the image recording unit 26. Further, language processing of re-assigning the language information LAN to the scene visualized in the determined image VD OR is performed. In the language processing, a text that describes a detailed situation corresponding to the specific language is generated, for example, using a VLM model (Vision Language Models). Thus, detailed language information LAN about a situation involved in the specific language included in the scene visualized in the determined image VD OR is generated. In a case where the detailed language information LAN is generated, in the update processing, the scene information SCN is updated by cooperation with the scene information recording unit 27.

[0049] 3. Effects

[0050] According to the embodiment described above, the reproduction processing of the scene visualized in the image VD OR is performed. In the reproduction processing, the virtual space VS that abstractly represents the real space RS visualized in the image VD OR is rendered, and the virtual person HM VR that abstracts the person HM visualized in the image VD OR is further rendered in the virtual space VS. Therefore, compared to a case where the image VD OR is directly used, it is possible to reduce the data size required for the playback of the scene of the image VD OR. In addition, since the virtual person HM VR is used, it is also possible to protect the privacy of the person HM.

[0051] Symbol Explanation:

[0052] 10… camera; 20… data processing device; 21… processing circuit; 22… storage device; 23… object recognition section; 24… verbalization section; 25… scene information generation section; 26… video recording section; 27… scene information recording section; 28… reproduction section; 29… scene information update section; 30… display device; 40… input device; HM… person; MO… moving object; SO… stationary object; NHM… moving object other than a person; HM_VR… virtual person; SO_VR… virtual stationary object; NHM_VR… virtual moving object; VD_OR, VD_VR… video; CAP… caption information; KWD… keyword information; LAN… language information; SAR… search information; SCN… scene information; REC_MO, REC_SO… recognition information.

Claims

1. A camera image management system for managing camera images, characterized in that, have: Storage device, storing the camera images; Processing circuits perform various processing steps; and Display device, outputs images. The processing circuit generates object recognition information reflected in the camera image by performing object recognition processing on the camera image stored in the storage device. The processing circuit generates language information reflecting the scene in the camera images by performing language processing on the camera images stored in the storage device. When the object recognition information includes person recognition information, the processing circuit generates scene information and stores it in the storage device. This scene information is information that associates the person recognition information with the linguistic information of the scene generated through the speech-based processing of the camera image of the identified person. The processing circuit performs scene reproduction processing based on the scene information stored in the storage device, which is reflected in the camera image. The reproduction process includes: An abstract image of the space reflected in the camera image is rendered based on the recognition information of static objects included in the scene information. Based on the character recognition information included in the scene information, the abstract image of the character reflected in the camera image is rendered onto the abstract image of the space. The title information of the scene is generated based on the language information of the scene included in the scene information; and The title information is appended to the abstract image of the space containing the rendered abstract image of the character, generating a reproduced image for output from the display device.

2. The camera image management system according to claim 1, characterized in that, When the object recognition information includes recognition information for dynamic objects other than people, the processing circuit adds the dynamic object recognition information to the scene information. The reproduction process also includes: Based on the recognition information of the dynamic objects included in the scene information, the abstract image of the dynamic objects is rendered onto the abstract image of the space.

3. The camera image management system according to claim 1 or 2, characterized in that, The processing circuit also detects a pre-defined specific language by referencing the language information of the scene included in the scene information. Upon detecting language information of the scene including the specific language, the processing circuit regenerates detailed language information of the scene reflected in the camera image by further language-based processing of the camera image, which is the source of the detected language information of the scene. The processing circuit updates the scene information, including information about the specific language, based on detailed language information of the scene.

4. The camera image management system according to claim 1 or 2, characterized in that, It also has an input device for receiving information. When retrieval information of a scene projected onto the camera image is input from the input device, the processing circuit further determines scene information that matches the retrieval information based on the retrieval information and the language information of the scene included in the scene information stored in the storage device. The reproduction process is performed based on scene information that matches the retrieved information.

5. The camera image management system according to claim 1 or 2, characterized in that, The abstract image of space includes a three-dimensional image that abstracts the space represented in the camera image. The abstract image of the person includes a three-dimensional image that abstracts the person as reflected in the camera image.

Images