Training data generation device, training data generation method, and program

The learning data generation device efficiently generates packet loss image data by dividing input images into packets and simulating packet loss, addressing the inefficiencies in existing methods by producing training datasets that accurately represent horizontal continuous defects, thus enhancing image restoration model accuracy.

WO2024247194A1PCT designated stage expired Publication Date: 2024-12-05NT T INC

Patent Information

Application Number
PCT/JP2023/020358
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Current methods for generating learning data for image restoration due to packet loss in uncompressed video transmission are inefficient, as they require constructing various packet loss rates and patterns for a large number of original images, and do not account for horizontal continuous defects, leading to a high packet loss rate that is not representative of conventional image restoration scenarios.

Method used

A learning data generation device and method that divides an input original image into packets, determines packet loss based on a specified rate, and generates packet loss image information with consecutive pixel missing in the horizontal direction, allowing for the creation of learning data for machine learning models to restore original images from packet loss images.

Benefits of technology

Enables efficient generation of learning data for image restoration, reducing the need for extensive environment setup and allowing for the creation of a large number of training datasets that reflect realistic packet loss patterns, thereby improving the accuracy of image restoration models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023020358_05122024_PF_FP_ABST
    Figure JP2023020358_05122024_PF_FP_ABST
Patent Text Reader

Abstract

A training data generation device 10 comprises a generation unit 21 that: divides an inputted original image in units of packets; determines whether each packet is lost on the basis of a packet loss rate; and generates packet loss image information, which corresponds to a packet determined to be lost and in which pixels where portions of the original image are contiguous in a horizontal direction are lost, as training data for machine learning.
Need to check novelty before this filing date? Find Prior Art

Description

Training data generation device, training data generation method, and program

[0001] The present disclosure relates to a training data generation device, a training data generation method, and a program.

[0002] SMPTE ST 2110 has been standardized as an uncompressed network video transmission method to replace the Serial Digital Interface (SDI), which has been widely used in broadcasting facilities. When monitoring video transmitted using SMPTE ST 2110, abnormalities caused by network transmission can be detected by network devices on the transmission path or by video receiving devices. For example, packet loss, where packets are lost during transmission, can be detected by an increase in the error packet counter in a network switch or by a missing RTP sequence number in a received packet.

[0003] However, when monitoring for abnormalities in the transmitted video essence itself, such as whiteout, where all pixels are solid white, or shading, where color unevenness occurs at the edges, it is necessary to extract the video essence from the received IP packets, reconstruct each frame, and visually check it.

[0004] From the viewpoint of scalability of the monitoring system, software implementation is desirable, but the data rate of uncompressed video flows is very high, and the data rate that can be received by software implementation is inferior to that of hardware implementation. As a method for reducing the reception rate, Patent Document 1 discloses a technology in which packet loss is intentionally generated in a video flow copied for monitoring purposes to reduce the data rate, and a low-resolution image is generated from the remaining packets after the packet loss and used for monitoring.

[0005] International Publication No. 2023 / 021547

[0006] A neural network can be used to restore the original image from a packet loss image, which is a frame-by-frame visualization of the remaining data contained in the remaining packets after packet loss. To train a neural network, a large number of original images and corresponding packet loss images must be prepared as training datasets (teaching data).

[0007] However, there are some challenges in obtaining the packet loss images required as a training dataset, as follows:

[0008] Conventionally, defects assumed in image restoration are those of specific objects, characters, or random defects, and horizontally continuous defects caused by packet loss that occurs in uncompressed video transmission are not taken into consideration.

[0009] In addition, there is a large difference between the data rate required for uncompressed video transmission and the data rate that can be received with current software implementations, and the difference becomes even larger as the video resolution increases, so the packet loss rate is higher than the packet loss rate assumed in conventional image restoration.

[0010] Therefore, it is possible to actually transmit each original image uncompressed, generate packet loss, and obtain packet loss images. However, this requires building an environment and obtaining various packet loss rates and loss patterns for a large number of original images, which requires a great deal of effort.

[0011] The present disclosure has been made in consideration of the above circumstances, and an object of the present disclosure is to provide a technology for easily generating training data for image restoration due to packet loss.

[0012] In order to achieve the above object, one aspect of the present disclosure is a training data generation device that includes a generation unit that divides an input original image into packets, determines whether each packet will be lost based on a packet loss rate, and generates, as training data for machine learning, packet loss image information in which a portion of the original image corresponding to a packet determined to be lost has consecutive pixels missing in the horizontal direction.

[0013] One aspect of the present disclosure is a training data generation method performed by a training data generation device, which divides an input original image into packets, determines whether each packet will be lost based on a packet loss rate, and generates packet loss image information as machine learning training data, in which a portion of the original image corresponding to a packet determined to be lost has horizontally consecutive missing pixels.

[0014] One aspect of the present disclosure is a program that causes a computer to function as the training data generation device.

[0015] According to the present disclosure, it is possible to provide a technology for easily generating learning data for image restoration due to packet loss.

[0016] FIG. 1 is a diagram showing an example of a packet loss image. FIG. 2 is a block diagram showing an example of the configuration of a training data generation device. FIG. 3 is a schematic diagram for explaining a mask generation unit and a synthesis unit. FIG. 4 is a flowchart showing an example of a packet loss image generation process. FIG. 5 is a diagram showing a schematic overview of machine learning using a packet loss image. FIG. 6 is a diagram showing a schematic process of estimating an original image from a packet loss image using a learning model. FIG. 7 is a diagram showing an example of a packet loss image with a biased packet loss rate. FIG. 8 is a diagram showing a function calculated by linear approximation from measured values. FIG. 9 is an example of a hardware configuration.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0018] <First Embodiment> This embodiment is an embodiment assuming a uniform packet loss rate across the entire image. In SMPTE ST 2110, pixel data of an image is divided, and packets are generated by storing the data uncompressed in the RTP payload, and the data is transmitted in packet units. Since each packet stores horizontally consecutive pixel data, packet loss of one packet causes horizontally consecutive pixel loss.

[0019] An example of a packet loss image obtained with a uniform packet loss rate across the entire image is shown in Figure 1. In the example shown, an original image 100 becomes a packet loss image 200 due to packet loss.

[0020] FIG. 2 is a block diagram showing an example of the configuration of the training data generating device 10 according to this embodiment.

[0021] The training data generation device 10 receives an original image as input and generates a packet loss image corresponding to the original image. The illustrated training data generation device 10 includes an input receiving unit 11, a generation unit 12, and a machine learning unit 13.

[0022] The input receiving unit 11 receives an input of an original image. The input receiving unit 11 may also receive predetermined input information. The input information may include a packet loss rate, a packet length, transmission flow information, a function indicating a bias in the packet loss rate, and the like.

[0023] The generation unit 12 divides the input original image into packets, determines whether each packet will be lost based on the packet loss rate, and generates a packet loss image in which consecutive pixels are missing horizontally from portions of the original image corresponding to packets determined to be lost, as learning data for machine learning. The packet loss image can be used as learning data for training a learning model for restoring the original image from the packet loss image.

[0024] Here, the generation unit 12 may include a mask generation unit 121 and a synthesis unit 122. The mask generation unit 121 generates a mask image in which areas where consecutive pixels are missing in the horizontal direction are masked. The synthesis unit 122 synthesizes the mask image with the original image to generate packet loss image information.

[0025] 3 is a schematic diagram illustrating the mask generation unit 121 and the synthesis unit 122. The mask generation unit 121 generates a loss mask based on the image size, packet loss rate, etc. of the original image 100. The synthesis unit 122 synthesizes the mask image M with the original image 100 to generate a packet loss image 200.

[0026] Specifically, the mask generation unit 121 generates the defect mask M according to the following equation.

[0027] Formula (1)

[0028] (x S ,y S ) is the starting coordinate of the Sth packet. S is the packet length, i.e., l S represents the number of pixels for each color component included in the payload. p is the packet loss rate. Here, 0 indicates a portion (packet) that you want to mask, and 1 indicates a portion (packet) that you do not want to mask.

[0029] The synthesis unit 122 multiplies (synthesizes) the input original image I with the loss mask M to generate a packet loss image I^.

[0030]

[0031] The generation unit 12 may not include the mask generation unit 121 and the synthesis unit 122, and may generate a packet loss image I^ with a uniform packet loss rate p from the original image I using the following equation without generating the loss mask M.

[0032] Formula (2)

[0033] (x S ,y S ) is the starting coordinate of the Sth packet. S is the packet length. p is the packet loss rate. Here, 0 indicates a missing part (packet), and I(x S ,y S ) indicates pixel data of the original image I without any loss.

[0034] The generating unit 12 may generate a packet loss image based on at least one of the packet length, the transmission flow information, and a function indicating the bias of the packet loss rate. The transmission flow information and the function will be described later.

[0035] In this embodiment, the generator 12 generates a packet loss image, but may also generate packet loss image information indicating the packet loss image. The packet loss image information may include at least one of a packet loss image, image features calculated from the packet loss image, and loss position information. The loss position information may be the aforementioned loss mask, or data including the starting coordinates of each packet and the position coordinates of lost packets, which are necessary for generating the packet loss image.

[0036] The machine learning unit 13 performs machine learning using the packet loss images generated by the generation unit 12, and generates a learning model 131. Note that the training data generation device 10 does not need to include the machine learning unit 13. In this case, another device (not shown) may perform machine learning using the packet loss images generated by the training data generation device 10, and generate the learning model 131.

[0037] 4 is a flowchart showing an example of a packet loss image generation process according to this embodiment. The training data generation device 10 determines the maximum number of epochs E and the number of original images N (S11). The maximum number of epochs E and the number of original images N may each be a predetermined number set in advance or may be a number input by the user.

[0038] S12A and S12B show epoch loop processing. The training data generation device 10 sets the initial value of variable e, which indicates the number of epochs, to 0 in S12A, adds 1 to variable e in S12B, repeats epoch loop processing if e<E, and ends processing when e=E. That is, the training data generation device 10 repeats epoch loop processing E times.

[0039] S13A and S13B represent loop processing of original images. In S13A, the training data generation device 10 sets the initial value of variable n, which indicates the number of original images, to 0, and in S13B adds 1 to variable n. If n<N, the training data generation device 10 repeats the loop of original images, and ends the processing when n=N. That is, the training data generation device 10 performs the processes of S14 to S16 for each of the N original images to generate a packet loss image.

[0040] The training data generation device 10 inputs an original image to be subjected to the loop, divides the original image into packets, and determines a loss pattern indicating whether each packet will be lost based on the packet loss rate (S14). Specifically, the training data generation device 10 sets the packet length and start coordinates when dividing and packetizing the original image in accordance with SMPTE ST 2110, and determines a loss pattern indicating whether each packet will be lost. That is, the training data generation device 10 determines whether each packet will be lost using the input packet loss rate.

[0041] As a specific example of determining whether each packet will be lost, the training data generating device 10 may generate a random number for each packet, and determine that packets will be lost if the random number itself or the output when the random number is input into a function exceeds a threshold calculated based on the packet loss rate, and determine that packets below the threshold will not be lost.

[0042] The training data generation device 10 generates a packet loss image based on the loss pattern (S15). That is, the training data generation device 10 generates a packet loss image in which consecutive pixels are missing horizontally from portions of the original image that correspond to packets determined to be lost. Specifically, the training data generation device 10 may generate a loss mask of Equation (1) based on the loss pattern determined in S14 and combine the loss mask with the original image to generate the packet loss image. Alternatively, the training data generation device 10 may generate a packet loss image from the original image using Equation (2) without generating a loss mask.

[0043] At this time, the training data generation device 10 may convert the generated packet loss image based on transmission flow information. For example, it is assumed that transmission flow information is input as input information. The transmission flow information includes, for example, a color mode, an image size, and the like.

[0044] For uncompressed video transmission, the color depth (bit count) and color mode (color format) of the color components must be specified in advance. For example, a transmission format commonly used in video production uses a 10-bit color mode called YCbCr422.

[0045] The color components are Y (luminance component) and CbCr (color difference components), and the horizontal size of CbCr is half that of Y. This is based on the human visual property that is less sensitive to changes in color difference images than to changes in luminance images.

[0046] In this embodiment, when generating a packet loss image, if a general RGB image or an image in another format is used as the original image, the training data generation device 10 may perform color mode conversion based on the transmission format specified in the transmission flow information. Furthermore, the training data generation device 10 may convert the packet loss image to an image size based on the transmission format specified in the transmission flow information.

[0047] The training data generation device 10 may cut out small patches from the generated packet loss image (S16). For example, if the N original images have different image sizes, the training data generation device 10 may cut out patches of the original image with a larger image size to match the original image with the smallest image size. Alternatively, the training data generation device 10 may cut out necessary portions of the packet loss image as patches. Note that the process of S16 may be omitted.

[0048] The training data generation device 10 performs the processes of S14 to S16 for each of the N original images for each epoch to generate N packet loss images. The training data generation device 19 then inputs the sets of the original images and the corresponding packet loss images as training data (training data, teacher data) to the machine learning unit 13 (or a machine learning device not shown).

[0049] The machine learning unit 13 performs machine learning using the input N pieces of learning data to generate a learning model 131 (S17). The learning model 131 may be, for example, a neural network, but is not limited to this. The machine learning unit 13 calculates the difference (loss) between the generated image generated by the learning model 131 from the packet loss image and the input original image, and feeds the difference (loss) back to the learning model 131.

[0050] 5 is a diagram schematically illustrating an overview of machine learning using a packet loss image. As shown in the figure, a packet loss image 200 generated by the generation unit 12 of the training data generation device 10 is input to a learning model 131, which generates a generated image 400 based on the packet loss image 200. The machine learning unit 13 calculates the difference between the generated image 400 generated by the learning model 131 and the original image 100, and performs machine learning by repeatedly feeding the difference back to the learning model 131.

[0051] In S12B, the training data generation device 10 adds 1 to e. If e<E, the process returns to S12A and repeats the loop process for the next epoch. At this time, in S14, the training data generation device 10 determines a packet length, packet loss rate, and loss pattern that differ from those of the previous training epoch. That is, the training data generation device 10 determines a packet length, packet loss rate, and loss pattern that differ for each epoch. This enables the generation of a more accurate training model 131 in this embodiment.

[0052] On the other hand, in S12B, 1 is added to e, and if e=E, the training data generation device 10 ends the processing shown in FIG.

[0053] 6 is a diagram schematically illustrating a process of generating a generated image 202 from a packet loss image 201 using a trained learning model 131 for which machine learning has been completed. In the illustrated example, when the packet loss image 201 is input to an estimation device 20 including the learning model 131, the learning model 131 outputs a generated image 202 corresponding to an original image estimated from the packet loss image 201.

[0054] Second Embodiment This embodiment assumes that the packet loss rate is biased in the horizontal direction of the original image. There are several patterns for storing pixel data in SMPTE ST 2110. However, when storing a fixed data size in the RTP payload from the left end of each line, a fractional portion may be generated at the right end of each line, resulting in packets of different sizes. This may result in a bias in which the packet loss rate is lowest at the left end of the original image and increases toward the right.

[0055] FIG. 7 shows an example of a packet loss image in which the packet loss rate is biased. In order to learn such biased packet loss, the generation unit 12 of the training data generation device 10 of this embodiment generates a packet loss image by changing the packet loss rate according to the position in the original image. Specifically, the generation unit 12 generates a packet loss image by changing the packet loss rate p(x S ) packet loss image. The training data generation device 10 of this embodiment is similar to the training data generation device 10 shown in FIG.

[0056] For example, the mask generating unit 121 of this embodiment generates the defect mask M according to the following equation.

[0057] Formula (3)

[0058] (x S ,y S ) is the starting coordinate of the Sth packet. S is the packet length, i.e., l S represents the number of pixels for each color component included in the payload. S ) is the starting coordinate of a packet (x S Specifically, p(x S ) is a function that derives the packet loss rate that changes in the horizontal direction. The synthesis unit 122 generates a packet loss image I^ by multiplying (synthesizing) the input original image I with the loss mask M.

[0059]

[0060] Furthermore, the generation unit 12 generates a function p(x S ) may be used to generate a packet loss image Î from the original image I by the following equation:

[0061] Formula (4)

[0062] (x S ,y S ) is the starting coordinate of the Sth packet. S is the packet length.

[0063] Note that the function p(x S The function information, such as the type, order, and parameters of the function that expresses the bias of (i.e., the variance of the training data), may be input from an external source, or part or all of the function information may be set by the training data generation device 10 as fixed values ​​or random numbers.

[0064] Furthermore, the training data generating device 10 calculates the function p(x S ) may be calculated from measurements taken in advance. That is, the original image may actually be transmitted uncompressed to cause packet loss, a packet loss image may be obtained, and the packet loss may be measured from the packet loss image.

[0065] Specifically, the generation unit 12 calculates a function p(x S ) is generated based on the measured packet loss data measured in advance, and the function p(x S ) and change the packet loss rate depending on the position of the original image to generate packet loss image information.

[0066] FIG. 8 shows a plurality of measured values ​​71 and a function p(x S ) = ax + b. The vertical axis of the graph shown in the figure is the packet loss rate, and the horizontal axis is the horizontal coordinate (x coordinate) of the packet start point. Measurement value 71 is obtained by transmitting an original image in advance and calculating the packet loss rate for each horizontal coordinate of the packet start point.

[0067] The generating unit 12 may linearly approximate the deviation of the packet loss rate as p(x)=ax+b, and estimate the parameters a and b from the measurement results using the least squares method of the following equation.

[0068]

[0069] n represents the nth original image, and p n (x S ) is the nth original image I n coordinate x S This represents the packet loss rate for the column.

[0070] The training data generation device 10 of the present embodiment described above includes a generation unit 12 that divides an input original image into packets, determines whether each packet will be lost based on a packet loss rate, and generates, as training data for machine learning, packet loss image information in which consecutive pixels are missing in the horizontal direction from a portion of the original image corresponding to a packet determined to be lost.

[0071] As a result, in this embodiment, training data for image restoration due to packet loss can be easily generated without actually transmitting each original image in an uncompressed manner, causing packet loss, and acquiring packet loss images. In other words, in this embodiment, it is possible to efficiently generate a large number of training data sets for restoring original images.

[0072] In addition, this embodiment makes it possible to generate a packet loss image that is close to an actual packet loss pattern, and to generate a group of packet loss images that reflect the packet loss patterns that occur in each environment.

[0073] The training data generation device 10 described above can use, for example, a general-purpose computer system as shown in Fig. 9. The illustrated computer system includes a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906. The memory 902 and the storage 903 are storage devices. In this computer system, the CPU 901 executes a predetermined program loaded on the memory 902, thereby realizing each function of the training data generation device 10.

[0074] The training data generation device 10 may be implemented on a single computer or multiple computers. The training data generation device 10 may also be a virtual machine implemented on a computer. The program of the training data generation device 10 can be stored on a computer-readable recording medium such as a HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or can be distributed via a network. The computer-readable recording medium is, for example, a non-transitory recording medium.

[0075] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0076] 10: Learning data generation device 11: Input reception unit 12: Generation unit 121: Mask generation unit 122: Synthesis unit 13: Machine learning unit 131: Learning model (neural network)

Claims

1. A learning data generation device comprising a generation unit that divides an input original image into packets, determines whether each packet will be lost based on a packet loss rate, and generates packet loss image information, as learning data for machine learning, in which parts of the original image corresponding to packets determined to be lost have consecutively lost pixels in the horizontal direction.

2. The learning data generation device according to claim 1, wherein the generation unit comprises: a mask generation unit that generates a mask image in which areas where consecutive pixels are missing in the horizontal direction are masked; and a synthesis unit that synthesizes the mask image with the original image to generate the packet loss image information.

3. The learning data generating device according to claim 1, wherein the generating unit generates the packet loss image information by changing a packet loss rate according to a position in the original image.

4. The learning data generating device according to claim 1, wherein the generating unit generates a function indicating a bias in the packet loss rate according to the position in the original image based on a previously measured packet loss value, and changes the packet loss rate according to the position in the original image in accordance with the function to generate the packet loss image information.

5. The learning data generating device according to claim 1, wherein the generating unit generates the packet loss image information based on at least one of a packet length, transmission flow information, and a function indicating a bias in the packet loss rate.

6. The learning data generation device according to claim 1, wherein the packet loss image information includes at least one of a packet loss image, an image feature calculated from the packet loss image, and information on a location of a loss.

7. A learning data generation method performed by a learning data generation device, which divides an input original image into packets, determines whether each packet will be lost based on a packet loss rate, and generates, as learning data for machine learning, packet loss image information in which a portion of the original image corresponding to a packet determined to be lost has consecutive pixels missing in the horizontal direction.

8. A program for causing a computer to function as the training data generation device according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Apparatus, method, and program for restoring moving picture information

    JP2005295054A

  • Method and apparatus for generating image restoration model, medium and program product

    US20230036338A1

Cited By

  • Image processing device, image processing system, image processing method, and image processing program

    WO2026186166A1