Encoding device, decoding device, encoding method, and decoding method

Abstract

To encrypt inference processing results of image data and information or the like used for mask processing, and embed them within video data.SOLUTION: An encoding device executes inference processing on image data containing personal information, generates metadata for the personal information included in the image data, specifies a type of mask processing to be executed in response to the inference processing, and executes mask processing on a region in which the personal information exists within the image data to generate masked image data on the basis of the metadata and the specified type of mask processing. The metadata indicates a region in which the personal information exists within the image data. The encoding device adds format-converted data generated by executing format conversion processing on encrypted metadata generated by encrypting the metadata and data used for the mask processing to video data generated by encoding the masked image data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an encoding device, a decoding device, an encoding method, and a decoding method. [Background technology]

[0002] In recent years, from the viewpoint of protecting personal information, images containing information that can identify individuals, such as a person's face or a vehicle license plate, i.e., personal information, are often subjected to image processing such as blurring or mosaic processing to make the individuals unidentifiable. These image processing methods are also called masking processes.

[0003] In the past, to protect personal information, image data editors would manually select areas that contained personal information and mask those areas. In recent years, there has been an increase in methods that use machine learning technology to detect areas that contain personal information and then mask those areas. However, while protecting personal information is important, there is also a need to be able to restore areas that have been masked.

[0004] Below, with reference to Figures 1 and 2 as prior art, we will explain a processing flow for performing masking processing by encrypting personal information data in image data, and a processing flow for restoring personal information in image data that has been masked in this way.

[0005] FIG. 1 is a diagram illustrating the process of an encoding device that performs masking on video data such as moving images. In this process, the video data is first decoded to extract image data. An inference process is performed on the extracted image data to identify the location of personal information. After the location of the personal information is identified, metadata including the frame position of the image data in the video data, the coordinates and attributes of the identified one or more pieces of personal information, image data of the one or more pieces of personal information (first personal information, second personal information), and image data in which the location where the personal information was present has been masked are generated. Each piece of personal information is encrypted. This process is repeated the same number of times as the number of pieces of image data. Multiple pieces of image data are encoded to generate video data, and the generated video data, encrypted personal information data, and metadata are associated and stored in storage. By performing this process, data containing personal information is securely stored, making it difficult to identify the personal information from the masked video data.

[0006] FIG. 2 is a diagram illustrating the process performed by a decoding device to restore masked video data by adding personal information. In this processing flow, first, metadata, personal information data, and video data that are associated with each other are read from storage, and the video data is decoded to generate image data. The decoding device references the metadata, decodes one or more pieces of personal information corresponding to the generated image data, and adds each piece of decoded personal information to a corresponding position in the image data to generate image data with the personal information combined. These processes are repeated the number of times equal to the number of pieces of decoded image data. Then, the image data with the personal information combined is encoded to generate video data. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2020-10365 Summary of the Invention [Problem to be solved by the invention]

[0008] As described above, even with conventional technology, it was possible to perform image processing (e.g., masking) on ​​video data containing personal information, such as blurring or mosaic processing, to prevent individuals from being identified, and then restore the masked areas at the desired timing.

[0009] However, in the conventional technology, in order to restore the masked video data, the masked video data, personal information data, and metadata must be associated and stored in storage, which poses a problem of requiring a large amount of storage capacity. Depending on the content of the video data, the amount of personal information data may be large, which further increases storage costs.

[0010] In the conventional technology, processing is performed on a frame-by-frame basis, so the video streaming process in the decoding device takes time, and there is a risk that the video data will not be able to perform in real time.

[0011] The present invention has been made to solve such problems, and aims to provide an encoding device that can encrypt metadata, which is the result of inference processing of image data, and information used for masking processing, etc., and include them in the video data itself, and a decoding device that can restore image data and video data including personal information by decrypting the video data output by such an encoding device. [Means for solving the problem]

[0012] In order to solve the above problem, an encoding device according to one aspect of the present invention includes: an inference means for performing an inference process on image data including personal information, identifying the personal information included in the image data, and generating metadata about the identified personal information; a rule generating means for specifying the type of masking process to be performed in response to the execution of the inference process; a mask processing means for performing mask processing on an area in the image data where personal information exists based on the metadata and the specified type of mask processing, to generate masked image data, wherein the metadata indicates the area in the image data where personal information exists; an encryption means for encrypting the metadata and the data used in the masking process to generate encrypted metadata; a conversion means for converting the encrypted metadata into a format to generate format-converted data to be added to the video data; means for encoding the masked image data to generate video data and adding the format-converted data to the video data; Equipped with. [Effects of the Invention]

[0013] According to the present invention, metadata, which is the result of inference processing of image data, and information used in masking processing, etc. can be encrypted and then included in the video data itself, thereby making it possible to significantly reduce storage costs compared to conventional technology.

[0014] According to the present invention, since the video data contains metadata, which is the result of inferring image data, and information used for masking, the video streaming processing time can be significantly reduced compared to the prior art. As a result, the decoding device can provide video with high real-time performance.

[0015] According to the present invention, the video data contains metadata, which is the result of inference processing of the image data, and information used for masking processing, etc., so that the decoding device can perform processing by reusing the inference results without receiving new data from the encoding device. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 10 is a diagram illustrating a process performed by an encoding device to perform a mask process on video data. [Figure 2] 10A and 10B are diagrams illustrating a process in a decoding device for restoring masked video data by adding personal information to the video data. [Figure 3] 1 is a diagram showing the configuration of an entire system including an encoding device 10 and a decoding device 20 according to the present invention. [Figure 4] FIG. 1 is a diagram illustrating a configuration of an encoding device 10 according to a first embodiment. [Figure 5] 1 is a system configuration diagram of an encoding device 10 according to the present invention. [Figure 6] FIG. 2 is a diagram illustrating a configuration of a decoding device 20 according to the first embodiment. [Figure 7] FIG. 1 is a system configuration diagram of a decoding device 20 according to the present invention. [Figure 8] FIG. 2 is a diagram illustrating an example of a processing flow executed by the encoding device 10. [Figure 9] FIG. 2 is a diagram illustrating an example of a processing flow executed by the decoding device 20. [Figure 10] 1 is a diagram illustrating an image of an exemplary process performed by the encoding device 10. FIG. [Figure 11] 10 is a diagram illustrating an image of an exemplary process performed by the decoding device 20. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0017] (Overall composition) Fig. 3 is a diagram showing the configuration of an entire system including an encoding device 10 and a decoding device 20 according to the present invention. The encoding device 10 transmits data to the decoding device 20 via a network 30 based on a request from the decoding device 20. The network 30 can be any network, such as the Internet, a dedicated line, a WAN, or a LAN, and is not particularly limited. Although Fig. 3 shows only one decoding device 20, there may be a plurality of such devices.

[0018] (Encoding and decoding device functions) To reduce storage costs, the encoding device of the present invention masks the areas of the image data where personal information is present, rather than extracting and separately storing personal information as in conventional techniques. The masking process performed by the encoding device can be thought of as a process of rearranging the pixel data where personal information is present according to a predetermined rule to make the personal information in the image data unreadable from the outside, or a process of blurring or mosaic-processing the areas where personal information is present and then cutting out and imaging the area. The former can be called a scrambling process, and the latter can be called an image cropping process. The encoding device encodes the image data to generate video data, and also encrypts and adds to the video data the positional information and attribute information (metadata) of the personal information in the masked image data, the scrambling rule information in the case of masking by scrambling, or the cropped image in the case of image cropping.

[0019] A decoding device according to the present invention extracts encrypted metadata from additional data (e.g., NAL units) associated with video data and decodes the video data to generate masked image data. By decrypting the encrypted metadata, the decoding device obtains position information and attribute information (metadata) of personal information in the masked image data, as well as information on the rule used for the masking process if the encoding device performed scrambling, or the cropped image data if the encoding device performed image cropping. The decoding device demasks the masked image data based on the obtained metadata and the rule information or the cropped image data. The decoding device can generate video data by encoding the demasked image data. The demasking process may also be referred to as descrambling. The decoding device can output the demasked image data or the encoded video data.

[0020] The encoding device of the present invention reduces the amount of data output and stored in storage compared to conventional technologies, thereby reducing storage costs. The encoding device and decoding device of the present invention do not require the storage and readout of image data from storage, as was required in conventional technologies, except for storing the distributed video data. This reduces the processing time required to restore personal information, enabling a system with high real-time performance in stream processing.

[0021] (First embodiment) An encoding device 10 and a decoding device 20 according to the first embodiment will be described. Fig. 4 is a diagram illustrating the configuration of the encoding device 10 according to the first embodiment. The encoding device 10 receives video data, performs the processing described below, and then outputs the video data to the decoding device 20. The encoding device 10 can distribute the video data based on a request from the decoding device 20.

[0022] The encoding device 10 includes a decoding unit 101, an inference unit 102, a rule generation unit 103, a mask processing unit 104, an encryption unit 105, a conversion unit 106, an encoding unit 107, and an insertion unit .

[0023] The decoding unit 101 generates image data by decoding video data received from an external device. A plurality of image data are generated from the video data. Some of the image data may contain personal information. The video data may be data in a stream format or data in a file format.

[0024] The inference unit 102 performs inference processing on each image data using a framework such as YOLO (You Only Look Once) to identify personal information contained in the image data. The encoding device 10 learns (trains) personal information that can identify individuals, such as human faces and vehicle license plates, and performs inference processing based on the learned information. The "learned information" may also be referred to as "training information."

[0025] When identifying personal information, the inference unit 102 acquires information on the position, attributes, and BBOX (bounding box) of the personal information within the image data. The position of the personal information within the image data is not limited, but may be coordinate information with the upper left corner of the image data as the base point. The attributes may be, for example, categories of information that can identify the personal information, such as a person's face or a vehicle license plate. The BBOX may indicate an area that includes the personal information when the position of the personal information within the image data is used as the base point.

[0026] The inference unit 102 generates metadata about the identified personal information. The metadata can include information for identifying the frame of the image data being processed, a timestamp, a label (attribute) for each piece of personal information, coordinate information, and BBOX information. Therefore, the metadata can indicate the area in the image data where the personal information exists.

[0027] The rule generation unit 103 can specify the type of masking to be performed by the mask processing unit 104 in response to the execution of the inference process by the inference unit 102. Possible types of masking include, but are not limited to, "scrambling" and "image clipping." The masking described herein refers to a process of reversibly processing data in an area of ​​image data containing personal information to render the original personal information invisible. It is sufficient that the personal information in the image data can be restored by a demasking process in the decoding device 20, as described below. When performing masking by scrambling, the rule generation unit 103 generates a rule to be used in the masking process according to a predetermined algorithm and provides the generated rule and information on the type of masking to be performed to the mask processing unit 104. Because the rule is generated in response to the execution of the inference process by the inference unit 102, it may differ for each piece of video data being processed. On the other hand, when performing masking by image clipping, the mask processing unit 104 does not generate the rule described above, but instead provides information on the type of masking to be performed to the mask processing unit 104.

[0028] The mask processing unit 104 can select the type of mask processing to perform based on the information on the type of mask processing provided by the rule generation unit 103. If the information on the type of mask processing indicates scrambling, the mask processing unit 104 performs masking on the area of ​​the image data where personal information exists, based on the metadata generated by the inference unit 102 and the rule generated by the rule generation unit 103, to generate masked image data. In this case, the masking (scrambling) process rearranges the arrangement of pixel data in the area according to a rule. The pixel data rearrangement process makes the original personal information invisible. On the other hand, if the information on the type of mask processing indicates image clipping, the mask processing unit 104 performs masking on the area of ​​the image data where personal information exists, based on the metadata generated by the inference unit 102, to generate masked image data. In this case, the masking (image clipping) process cuts out the image data of the area indicated by the metadata and performs blurring, mosaic, or the like on the area. The original personal information becomes invisible through blurring, mosaic, or the like.

[0029] The encryption unit 105 encrypts the metadata generated by the inference unit 102 and the information on the rule generated by the rule generation unit 103 if a scrambling process has been performed, or the cut-out image data if an image cutting process has been performed, to generate encrypted metadata. That is, the encrypted metadata includes metadata about the personal information generated by the inference unit 102 and the information on the rule generated by the rule generation unit 103 and used in the masking process by the masking processing unit 104, or the image data cut out by the image cutting process. The encryption process may be performed based on an encryption method such as attribute-based encryption (ABE), for example, but is not limited to a specific encryption method.

[0030] The conversion unit 106 performs a format conversion process on the encrypted metadata generated by the encryption unit 105 to generate format-converted data that can be added to video data. The format-converted data may be, for example, a NAL (Network Abstraction Layer) unit. Data stored in the RBSP (Raw Byte Sequence Payload) of the NAL unit may be based on the encrypted metadata generated by the encryption unit 105.

[0031] The encoding unit 107 generates video data by encoding the multiple pieces of image data (masked image data) that have been masked by the masking processing unit 104. The inserting unit 108 adds the format-converted data, for example, NAL units, generated by the converting unit 106 to the video data generated by the encoding unit 107, and outputs the video data after the addition processing. The inserting unit 108 stores the video data after the addition processing in a storage within the encoding device 10, and transmits it to the decoding device 20.

[0032] (System Configuration) 5 is a system configuration diagram of an encoding device 10 according to the present invention. Similar to a general computer, encoding device 10 comprises a processor 11, a main memory unit 12, an auxiliary memory unit 13, an IF unit 14, and an output unit 15, which are interconnected by a bus 17 or the like. Encoding device 10 also comprises video data 16 to be distributed in the form of storage means such as a file / database in auxiliary memory unit 13.

[0033] The processor 11, also called a central processing unit (CPU), controls each component of the encoding device 10 and performs data calculations, and also reads various programs stored in the auxiliary storage unit 13 into the main storage unit 12 and executes them. The main storage unit 12, also called a main memory, can store various received data, computer-executable instructions, and data after calculation processing based on those instructions. The auxiliary storage unit 13 is a storage device such as a hard disk drive (HDD) or solid-state drive (SSD), and is used for long-term storage of data and programs.

[0034] 5 illustrates an embodiment in which the processor 11, main memory 12, and auxiliary memory 13 are provided in the same computer, but in another embodiment, the encoding device 10 can be configured to realize parallel distributed processing by a plurality of computers by using a plurality of processors 11, main memory 12, and auxiliary memory 13. In another embodiment, a plurality of servers for the encoding device 10 can be installed, and a single auxiliary memory 13 can be shared by the plurality of servers.

[0035] The IF unit 14 plays the role of an interface (IF) when sending and receiving data to and from other systems and devices, and provides an interface for accepting various commands and input data (various masters, tables, etc.) from a system operator. The output unit 15 provides a display screen for displaying processed data and printing means for printing the data.

[0036] The video data for distribution 16 stores the video data output after the above-described processing by the encoding device 10. Based on a request from the decoding device 20, the encoding device 10 can re-read data that has been once distributed from the video data for distribution 16 and transmit it to the decoding device 20.

[0037] 6 is a diagram illustrating the configuration of the decoding device 20 according to the first embodiment. The decoding device 20 receives the video data output by the encoding device 10, performs the processing described below, and then outputs the image data or video data (for example, to a user terminal).

[0038] The decoding device 20 may include an extraction unit 201, a decoding unit 202, a decoding unit 203, a restoration unit 204, an encoding unit 205, and an insertion unit 206. When the decoding device 20 outputs image data, the processing by the encoding unit 205 and the insertion unit 206 is not performed.

[0039] The extraction unit 201 stores the video data received from the encoding device 10 in a storage in the decoding device 20, and extracts format-converted data (additional data) from the video data. As described above, the video data received from the encoding device 10 includes format-converted data such as NAL units.

[0040] The decoding unit 202 decodes the received video data to generate image data.

[0041] The decoding unit 203 extracts encrypted metadata from the format-converted data (e.g., NAL units) extracted by the extraction unit 201. The decoding unit 203 decodes the encrypted metadata to extract the metadata and rule information or extracted image data. That is, if the masking process performed by the encoding device is a scrambling process, the rule information is extracted by the decoding process. On the other hand, if the masking process is an image extraction process, the extracted image data is extracted by the decoding process. The extracted metadata includes information identifying the frame of the image data, a timestamp, labels (attributes) for each piece of personal information, coordinate information, and BBOX (bounding box) information. That is, the metadata indicates the area in the image data where the personal information exists. The extracted rule includes rearrangement information that was the basis for the masking process performed by the encoding device 10. The extracted image data is image data that existed in the area corresponding to the personal information when the masking process performed by the encoding device 10 was performed.

[0042] The restoration unit 204 performs a demasking process on the area in each image data where personal information exists, based on the metadata extracted by the decoding unit 203 and the rule or the extracted image data, to generate demasked image data. Because the rule includes rearrangement information for the pixel data that was the basis for the masking process, the restoration unit 204 can use the rule information to perform a demasking process that rearranges the pixel data in the reverse order of the masking process performed by the encoding device 10. On the other hand, when the extracted image data has been extracted, the restoration unit 204 can perform a demasking process that adds corresponding image data to the area indicated by the metadata (i.e., the area that has been blurred or pixelated). The demasking process restores the personal information in the image data.

[0043] The encoding unit 205 can generate video data by encoding the image data that has been demasked by the restoration unit 204. The insertion unit 206 can insert additional data such as NAL units into the video data generated by the encoding unit 205 and generate video data to be provided to a user.

[0044] (System Configuration) 7 is a system configuration diagram of a decoding device 20 according to the present invention. Similar to a general computer, the decoding device 20 includes a processor 21, a main memory unit 22, an auxiliary memory unit 23, an IF unit 24, and an output unit 25, which are interconnected by a bus 27 or the like. The decoding device 20 can include video data 26 in the form of storage means such as a file or database in the auxiliary memory unit 23. In one embodiment of the present invention, when the decoding device 20 displays the restored video data on a monitor or distributes it via RTSP (Real Time Streaming Protocol), the data does not need to be stored in the auxiliary memory unit 23.

[0045] The processor 21, main memory unit 22, auxiliary memory unit 23, IF unit 24, and output unit 25 perform the same functions as the processor 11, main memory unit 12, auxiliary memory unit 13, IF unit 14, and output unit 15, respectively, described with reference to Figure 5, so detailed explanations will be omitted.

[0046] The video data 26 may store video data received from the encoding device 10 .

[0047] (Processing flow executed by the encoding device 10) Fig. 8 illustrates an example of a processing flow performed by the encoding device 10. Fig. 10 illustrates an image of an exemplary process performed by the encoding device 10, but the embodiment of the present invention is not limited thereto. For example, the video data is not limited to stream format data.

[0048] In S801, the encoding device 10 decodes video data received from an external device. For example, the encoding device 10 decodes video data (Streamed Video Data) as shown in FIG. 10. The decoded video data is divided into multiple pieces of image data (Raw Video Data). Some of the image data may include personal information.

[0049] In S802, the encoding device 10 uses a framework such as YOLO (You Only Look Once) to perform inference processing on each image data based on information learned by the encoding device 10, and identifies personal information contained in the image data. As illustrated in FIG. 10, the image data may include one or more pieces of personal information such as "Attribute 1" and "Attribute 2."

[0050] The encoding device 10 generates metadata about the personal information identified by the inference process. The metadata can include information identifying the frame of the image data being processed, a timestamp, a label (attribute) for each piece of personal information, coordinate information, and BBOX (bounding box) information.

[0051] In S803, in response to the execution of the inference process in S802, the encoding device 10 specifies the type of masking process to be executed. This specification may be determined based on the type of video data input to the encoding device 10, the identification information of the decoding device 20 to which the processed video data is to be distributed, and the like. Possible types of masking processes include, but are not limited to, scrambling and image cropping. If the specified type of masking process indicates scrambling, the encoding device 10 generates a rule for performing the masking process (scrambling), which will be described later, based on a predetermined algorithm. Because the rule is generated in response to the execution of the inference process, it may differ for each piece of video data to be processed.

[0052] In S804, the encoding device 10 performs masking on the area of ​​the image data where personal information exists, based at least on the metadata generated in S802. If the type of masking is scrambling, the encoding device 10 can perform masking further based on the rule generated in S803. In the case of scrambling, the masking is a process of rearranging the pixel data in the area according to the rule, and in the case of image cropping, the masking is a process of pasting a blurred image or a mosaic image onto the original image in the area. In either case, it is sufficient that the original personal information is not visible. As illustrated in FIG. 10, attributes 1 and 2 of the image data (raw video data) after the masking process are processed so that the original image cannot be recognized. Note that the image cropping is performed before the masking process, and the image data of the area where personal information exists within the image data is cropped before the masking process.

[0053] In S805, the encoding device 10 generates video data by encoding the plurality of image data after the masking process. The example in Fig. 10 shows that the plurality of image data after the masking process (Raw Video Data) is encoded to generate video data (Streamed Video Data).

[0054] In S806, the encoding device 10 encrypts the metadata generated in S802 and the information on the rule generated in S803 if a scrambling process has been performed, or the cut-out image data if an image cutting process has been performed, to generate encrypted metadata. As illustrated in Fig. 10, the encrypted metadata may include metadata about personal information and information on the rule used in the masking process or the image data cut out by the image cutting process. The encryption may be performed based on an encryption method such as attribute-based encryption (ABE), but is not limited to a specific encryption method.

[0055] In S807, the encoding device 10 performs a format conversion process on the encrypted metadata to generate format-converted data that can be added to video data. The format-converted data may be in any format that can be added to video data, such as, but not limited to, NAL (Network Abstraction Layer) units of the H.264 or H.265 standard.

[0056] In S808, the encoding device 10 adds (inserts) the format-converted data to the video data encoded in S805, stores the video data in the video data for distribution 16, and outputs the video data to the predetermined distribution destination, the decoding device 20. The encoding device 10 can distribute the video data based on a request from the decoding device 20.

[0057] (Processing flow executed by the decoding device 20) Fig. 9 illustrates an example of a processing flow performed by the decoding device 20. Fig. 11 illustrates an image of an exemplary process performed by the decoding device 20, but the embodiment of the present invention is not limited thereto. For example, the video data is not limited to stream format data.

[0058] In S901, the decoding device 20 stores the video data received from the encoding device 10 in a storage (video data 26) within the decoding device 20, and extracts the format-converted data (additional data) from the video data. As illustrated in Fig. 11, the received video data includes the format-converted data described above.

[0059] In S902, the decoding device 20 decodes the received video data. For example, as illustrated in Fig. 11, the received video data (Streamed Video Data) is divided into a plurality of image data (Raw Video Data).

[0060] In S903, the decoding device 20 extracts encrypted metadata from the format-converted data extracted in S901 and decrypts the extracted encrypted metadata to extract the metadata and rule information or extracted image data. For example, the format-converted data corresponds to an NAL unit in the H.264 or H.265 standard, and the RBSP contains encrypted metadata. The example in FIG. 11 illustrates that the metadata and rule information or extracted image data are extracted from the encrypted metadata. The extracted metadata includes information identifying the frame of the associated image data, a timestamp, a label (attribute) for each piece of personal information, coordinate information, and BBOX (bounding box) information. In other words, the metadata indicates the area in the image data where the personal information exists. The extracted rule may include rearrangement information that served as the basis for the masking process. Alternatively, the extracted image data may be image data that existed in the area corresponding to the personal information during the masking process performed by the encoding device 10. The extracted image data is extracted by decrypting the encrypted image data from the encrypted metadata.

[0061] At S904, the decoding device 20 performs a demasking process on each piece of image data decoded at S902 using the metadata extracted at S903 and the rule or the extracted image data to generate demasked image data. That is, the decoding device 20 can identify areas of the image data where personal information exists based on the metadata and perform a pixel data rearrangement process based on the rule. The rearrangement process performed by the decoding device 20 is a process performed in the reverse order of the rearrangement process performed by the encoding device 10. Alternatively, the decoding device 20 can identify areas of the image data where personal information exists based on the metadata and add corresponding extracted image data to the identified areas. The decoding device 20 can output the demasked image data and provide it to a user.

[0062] In S905, the decoding device 20 can generate video data by encoding the image data that has been demasked in S904. For example, in the example of Fig. 11, video data (Streamed Video Data) is generated based on a plurality of image data (Raw Video Data) that have been demasked.

[0063] In S906, the decoding device 20 inserts additional data such as NAL units into the video data. The example in Fig. 11 shows that the additional data has been inserted into the video data (Streamed Video Data). The decoding device 20 can provide the video data after the additional data has been inserted to any user terminal.

[0064] (Second embodiment) In the above embodiment, the encoding device 10 performs masking on all personal information contained in the image data identified by the inference process. However, the encoding device 10 may be configured to select personal information to be masked in the image data, such as masking only a person's face, based on predetermined masking process selection information. Similarly, the decoding device 20 may be configured to select personal information to be demasked based on predetermined demasking process selection information. By adopting such a configuration, the encoding device 10 can select the type of personal information to be masked, and the decoding device 20 can restore only the desired type of personal information, even if all types of personal information have been masked.

[0065] In the above embodiment, the masking process (scrambling process) refers to a process of rearranging pixel data in an area of ​​image data where personal information exists according to rule information. In the present invention, pixel values ​​may additionally or alternatively be calculated based on second rule information. The second rule information may be generated by the rule generation unit 103 in response to the execution of the inference process by the inference unit 102. In such a case, the masking process unit 104 performs masking by changing the pixel values ​​of each pixel in the area of ​​image data where personal information exists according to the second rule information, based on the metadata generated by the inference unit 102 and the second rule generated by the rule generation unit 103, to generate masked image data.

[0066] The encoding device 10 can distribute video data in response to a request from the decoding device 20, and can select whether or not to mask personal information depending on the destination decoding device 20. For example, masking is not necessary as long as the data is used within the same workplace (e.g., a police station), but masking is necessary when the data is sent outside the workplace. Therefore, based on the identification information of the decoding device 20 that has sent the request, the encoding device 10 can determine whether or not to mask the data and, if so, what attributes of personal information to mask.

[0067] (summary) The encoding device of the present invention decodes video data, identifies an area in the image data that contains personal information, rearranges the array of pixel data in that area according to a predetermined rule, or cuts out the image data of that area and performs a blurring process on that area, encrypts metadata related to the personal information and the rule or the cut-out image data, converts the format of the encrypted data, and outputs video data in which the format-converted data is stored in association with the image data. In this invention, information on the rearrangement of the pixel data of an image or the cut-out image data is stored in association with the video data, and its configuration differs from that of the invention disclosed in Patent Document 1.

[0068] In the present invention, encrypted data is inserted into the format-converted data (e.g., NAL units), making it possible to transfer metadata securely and at low cost. As a result, it is no longer necessary to use storage to store various data as in conventional technology, thereby reducing storage costs.

[0069] Although the principles of the present invention have been described above with reference to exemplary embodiments, it will be understood by those skilled in the art that various modifications in configuration and detail can be made without departing from the spirit of the present invention. That is, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, or a storage medium. [Explanation of symbols]

[0070] 10 Encoding device 20 Decryption device 30 Network 11, 21 processors 12, 22 Main memory 13, 23 Auxiliary storage 14, 24 IF Section 15, 25 Output section 16. Video data for distribution 26 Video data 101 Decoder 102 Reasoning part 103 Law generation part 104 Mask processing section 105 Encryption section 106 Conversion unit 107 Encoding section 108 Insertion section 201 Extraction part 202 Decoding section 203 Decoding Unit 204 Restoration Department 205 Encoding section 206 Insertion section

Claims

1. an inference means for performing an inference process on image data including personal information, identifying the personal information included in the image data, and generating metadata about the identified personal information; a rule generating means for specifying the type of masking process to be performed in response to the execution of the inference process; a mask processing means for performing mask processing on an area in the image data where personal information exists based on the metadata and the specified type of mask processing, to generate masked image data, wherein the metadata indicates the area in the image data where personal information exists; an encryption means for encrypting the metadata and the data used in the masking process to generate encrypted metadata; a conversion means for converting the encrypted metadata into a format to generate format-converted data to be added to the video data; means for encoding the masked image data to generate video data and adding the format-converted data to the video data; An encoding device comprising:

2. 2. The encoding device of claim 1, wherein when the specified type of masking indicates scrambling, the rule generating means further generates a rule to be used in the masking, and the masking is a process of rearranging the arrangement of pixel data in the region according to the rule.

3. The encoding device of claim 1, wherein when the specified type of masking process indicates an image cutting-out process, the masking process is a process of cutting out image data of personal information present in the area and performing blurring or mosaic processing on the area.

4. 2. The encoding device of claim 1, wherein said inference means performs said inference process based on training information learned by said encoding device.

5. The encoding device according to claim 1 , wherein the metadata includes information for identifying a frame of the image data, a time stamp, attribute information relating to the personal information, coordinate information, and BBOX information.

6. 2. The encoding device according to claim 1, further comprising: decoding means for generating said image data by decoding video data received from an external device.

7. 2. The encoding device according to claim 1, wherein personal information to be masked within the image data is selected based on predetermined masking selection information.

8. the rule generating means generates a second rule to be used in a masking process in response to the execution of the inference process; the mask processing means performs mask processing based on the metadata and the second rule by changing pixel values ​​of each pixel in an area where personal information exists in the image data based on information of the second rule, thereby generating mask-processed image data. The encoding device of claim 1.

9. performing an inference process on image data including personal information to identify personal information contained in the image data and generate metadata about the identified personal information; In response to the inference process being performed, specifying a type of masking process to be performed; Based on the metadata and the specified type of masking, a masking process is performed on an area in the image data where personal information exists, thereby generating masked image data, wherein the metadata indicates an area in the image data where personal information exists; encrypting the metadata and the data used in the masking process to generate encrypted metadata; performing a format conversion process on the encrypted metadata to generate format-converted data to be added to the video data; encoding the masked image data to generate video data, and adding the format-converted data to the video data; An encoding method comprising:

10. A decoding device including a communication unit for communicating with an external network, a storage unit, and a control unit, The control unit storing video data received from the encoding device via the communication unit in the storage unit, and decoding the video data to generate image data; Decrypting encrypted metadata included in the first format data extracted from the received video data to extract metadata and data used for demasking processing, the metadata indicating an area in the image data where personal information exists; Based on the extracted metadata and data used for the demasking process, perform a demasking process on the area in the image data where personal information exists to generate demasked image data; A decoding device that performs

11. the data used for the demasking process is the law used for the masking process performed by the encoding device; The decoding device of claim 10, wherein the demasking process is a process of rearranging pixel data in an area of ​​the image data identified by the metadata where personal information exists in the reverse order of the rearrangement order indicated by the rule.

12. the data used in the demasking process is image data extracted in the masking process executed by the encoding device; The decoding device according to claim 10 , wherein the demasking process is a process of adding the corresponding cut-out image data to an area in the image data identified by the metadata where personal information exists.

13. The decryption device according to claim 10 , wherein the control unit selects personal information to be demasked based on predetermined demasking process selection information, and performs demasking on an area where the selected personal information exists.

14. A decoding method performed by a decoding device, comprising: the decoding device includes a communication unit for communicating with an external network, a storage unit, and a control unit; The decoding method comprises: the control unit stores the video data received from the encoding device via the communication unit in the storage unit, and decodes the video data to generate image data; the control unit decrypts encrypted metadata included in the first format data extracted from the received video data to extract the metadata and data used for demasking processing, the metadata indicating an area in the image data where personal information exists; The control unit performs a demasking process on an area in the image data where personal information exists based on the extracted metadata and data used for the demasking process to generate demasked image data; A decoding method comprising:

Images