Video signal processing method and video signal processing device

Abstract

To provide a video signal processing method capable of performing video distribution using data different from video data.SOLUTION: A video signal processing method according to an embodiment includes: a step for receiving first data; a step for generating second data by converting the received first data to an RGB value; a step for receiving a first video signal including an RGB value for each pixel; a step for generating a second video signal by substituting an RGB value in a first region within the received first video signal by an RGB value of the second data; and a step for outputting the second video signal.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a video signal processing method and a video signal processing device relating to processing of a video signal. [Background technology]

[0002] Patent Document 1 describes a digital watermark information embedding device that outputs an image signal in which digital watermark information is embedded.

[0003] Patent Document 2 describes a data information embedding device and a playback device. The data information embedding device generates watermark information from data information. The data information embedding device embeds the generated digital watermark information into video and audio signals. The data information embedding device outputs the digital watermark information and the video and audio signals.

[0004] Patent Document 3 describes a display control device. The display control device generates information such as the distance from a person at the time of shooting from image data as metadata. The display control device encodes the video and metadata. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 3587168 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-130374 [Patent Document 3] Patent Publication No. 2011-221844 Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, there is a demand to distribute data (hereinafter referred to as data A) that is different from the video signal of a video along with the distribution of the video signal. However, existing video distribution platforms only distribute video signals of videos, and there are cases in which they are unable to distribute data A.

[0007] An embodiment of the present invention aims to provide a video signal processing method that can distribute data different from the video signal along with the video signal, even on an existing video distribution platform. [Means for solving the problem]

[0008] A video signal processing method according to an embodiment of the present invention includes: Accept the first data, generating second data by converting the received first data into RGB values; receiving a first video signal including RGB values ​​for each pixel; generating a second video signal by replacing RGB values ​​of pixels in a first region of the received first video signal with RGB values ​​of the second data; The second video signal is output. [Effects of the Invention]

[0009] According to the video signal processing method of one embodiment of the present invention, even on an existing video distribution platform, it is possible to distribute a video signal as well as data different from the video signal. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a video signal processing device 20 and a terminal 30 that execute a video signal processing method according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of connections between the video signal processing device 20, the terminal 30, and the server 40. [Figure 3] FIG. 3 is a diagram showing the concept of two or more frames. [Figure 4] FIG. 4 is a flowchart showing an example of processing by the video signal processing device 20. [Figure 5] FIG. 5 is a diagram showing data movement in the video signal processing device 20. As shown in FIG. [Figure 6] FIG. 6 is a diagram showing the concept of data processing in the video signal processing device 20. As shown in FIG. [Figure 7] FIG. 7 is a flowchart showing an example of the decoding process of the second video signal Vd2. [Figure 8] FIG. 8 shows an area a1 containing 4×4 pixels. [Figure 9] FIG. 9 is a diagram showing an example of conversion of first data D1 including identifier data ID. [Figure 10] FIG. 10 is a flowchart showing an example of the decryption process executed by the terminal 30 in the third modification. [Figure 11] FIG. 11 is a diagram showing an example of processing in the video signal processing device 20d. [Figure 12] FIG. 12 is a flowchart showing an example of the decryption process executed by the terminal 30 in the fourth modification. [Figure 13] FIG. 13 is a diagram showing an example of processing in the video signal processing device 20e. [Figure 14] FIG. 14 is a diagram showing an example of processing in the terminal 30. [Figure 15] FIG. 15 is a diagram showing an application example 1 of the video signal processing devices 20, 20a to 20e. DETAILED DESCRIPTION OF THE INVENTION

[0011] (First embodiment) The video signal processing method according to the first embodiment will be described below with reference to the drawings. Fig. 1 is a block diagram showing an example of the configuration of a video signal processing device 20 that executes the video signal processing method. Fig. 2 is a block diagram showing an example of the connection between the video signal processing device 20, a terminal 30, and a server 40. Fig. 3 is a diagram showing the concept of two or more frames.

[0012] The video signal processing device 20 is a device that generates a video signal. In this embodiment, the video signal includes data for displaying a video on a video playback device. In this embodiment, the video signal also includes a video signal obtained by decoding a signal transmitted in a compressed state. The display on the display of the video playback device changes based on the video signal. Therefore, the video signal includes, for example, RGB value data for each pixel. Furthermore, since the video signal is a signal related to the playback of a moving image, it includes two or more frames. The video playback device then plays the moving image by outputting each of the two or more frames in sequence to the display.

[0013] As shown in FIG. 1, the video signal processing device 20 includes a display 200, a processing unit 201, a communication interface 202, a user interface 203, a flash memory 204, and a RAM (Random Access Memory) 205.

[0014] The flash memory 204 stores various programs, such as a program for operating the video signal processing device 20 or an application program for generating a video signal.

[0015] The RAM 205 temporarily stores the predetermined program stored in the flash memory 204 .

[0016] The processing unit 201 is configured by a CPU (Central Processing Unit) and controls the operation of the video signal processing device 20. Specifically, the processing unit 201 reads out programs stored in a flash memory 204 into a RAM 205 to execute various operations.

[0017] The communication interface 202 communicates with devices different from the video signal processing device 20 (hereinafter referred to as external devices) via a communication line. The video signal processing device 20 and the external devices are connected to each other wirelessly, for example, via Wi-Fi (registered trademark) or Bluetooth (registered trademark), or by wire. The communication interface 202 is, for example, a USB, an HDMI (registered trademark), or a network interface. The communication interface 202 corresponds to an output unit in the present invention.

[0018] The display 200 displays various information based on the operation of the processing unit 201. The display 200 is, for example, a liquid crystal display or an organic EL display.

[0019] The user interface 203 accepts operations on the video signal processing device 20 from a user of the video signal processing device 20. The user interface 203 is, for example, a keyboard, a mouse, or a touch panel.

[0020] The video signal processing device 20 as described above is, for example, a smartphone or a PC.

[0021] 2, the video signal processing device 20 is communicably connected to a terminal 30 and a server 40 via a communication line 50. The terminal 30 is an example of a device different from the video signal processing device 20. In this case, the video signal processing device 20 communicates with the terminal 30 and the server 40 via a communication interface 202. The communication line 50 is, for example, an Internet line.

[0022] The communication line 50 does not necessarily have to be an Internet line. The video signal processing device 20, the terminal 30, and the server 40 may communicate with each other via a private network that is not connected to the Internet.

[0023] The server 40 receives and stores the video signal generated in the video signal processing device 20. Specifically, the server 40 receives the video signal from the video signal processing device 20 via a communication line 50. The server 40 stores the received video signal. The server 40 also constitutes a video distribution platform.

[0024] Terminal 30 receives and plays back a video signal by connecting to server 40. Specifically, terminal 30 receives and plays back a video signal distributed by server 40, which is a video distribution platform. This allows a user of terminal 30 to view video related to the video signal. Such a terminal 30 is, for example, a smartphone or a PC.

[0025] The video signal processing device 20 according to this embodiment embeds data different from the video signal itself into the video signal. Specifically, the video signal processing device 20 generates a second video signal Vd2 by embedding data different from the first video signal Vd1 into the first video signal Vd1 input from a video camera or the like. The first video signal Vd1 is an example of a video signal. Accordingly, the first video signal Vd1 includes RGB values ​​for each pixel. The first video signal Vd1 also includes two or more frames. For example, as shown in FIG. 3, the first video signal Vd1 includes a first frame 300 and a second frame 301. For ease of understanding, directions are defined below as shown in FIG. 3. Specifically, the direction in which regions a1 to a6 are aligned in the first frame 300 is defined as the X-axis direction. The direction perpendicular to the X-axis direction in the first frame 300 is defined as the Y-axis direction.

[0026] The data different from the first video signal Vd1 is, for example, lighting data. Specifically, lighting data is data for controlling lighting. For example, a lighting device changes the brightness, color, etc. of the lighting based on the lighting data. Hereinafter, the data different from the first video signal Vd1 described above will be referred to as first data D1. Therefore, in this embodiment, the video signal processing device 20 embeds the first data D1 of lighting data in the first video signal Vd1.

[0027] The process by which the video signal processing device 20 generates the second video signal Vd2 will be described in more detail below with reference to the drawings. Fig. 4 is a flowchart showing an example of the process by the video signal processing device 20. Fig. 5 is a diagram showing the concept of data movement in the video signal processing device 20. Fig. 6 is a diagram showing the concept of data processing in the video signal processing device 20. For example, when an application program related to video signal processing is executed, the video signal processing device 20 starts the process of generating the second video signal Vd2 (FIG. 4: START).

[0028] First, the processing unit 201 receives the first data D1 (FIG. 4: step S11). Specifically, in this embodiment, the communication interface 202 receives the first data D1 from a lighting device control device or the like (see FIG. 5). Then, the processing unit 201 receives the first data D1 from the communication interface 202. Note that the processing unit 201 may generate the first data D1 based on a lighting device control signal pre-recorded in the video signal processing device 20, for example. Alternatively, the processing unit 201 may receive a lighting device control signal pre-recorded in a lighting device control device or the like, and generate the first data D1 based on the received lighting device control signal.

[0029] Next, the processing unit 201 receives the first video signal Vd1 (FIG. 4: step S12). For example, the communication interface 202 receives the first video signal Vd1 from a video camera for capturing moving images (see FIG. 5). Then, the processing unit 201 receives the first video signal Vd1 from the communication interface 202.

[0030] Next, the processing unit 201 generates second data D2 by converting the received first data D1 into RGB values ​​(FIG. 4: step S13). For example, the processing unit 201 converts the byte values ​​of the first data D1 into bit values, as shown in FIG. 6. In the example shown in FIG. 6, the processing unit 201 converts the byte values ​​"0x11, 0x13" of the first data D1 into bit values ​​"00010001, 00010011". This allows the processing unit 201 to obtain the first data D1, which is a bit string.

[0031] After the conversion, the processing unit 201 divides the bit string of the first data D1 into groups of three bits. In the example shown in Fig. 6, the bit string of the first data D1 is "00010001, 00010011". Therefore, the processing unit 201 obtains one or more bit strings divided into groups of three bits, "000, 100, 010, 001, 001, 100". When one or two bits remain, the processing unit 201 adds a 0 bit to obtain a bit string of three bits.

[0032] Then, the processing unit 201 converts each of the bit strings divided into three bits into an RGB value. For example, as shown in FIG. 6, the processing unit 201 converts the three-bit string "010" into an RGB value of RGB=(0,255,0). In this embodiment, the value of the first bit of the three bits corresponds to the R value in the RGB value. Specifically, when the first bit of the three bits is "1", the processing unit 201 obtains a conversion result of R=255. On the other hand, when the value of the most significant bit of the three bits is "0", the processing unit 201 obtains a conversion result of R=0. Similarly, the value of the second most significant bit of the three bits corresponds to the G value in the RGB value. Similarly, the value of the least significant bit of the three bits corresponds to the B value in the RGB value.

[0033] The processing unit 201 generates second data D2 by converting all bit strings separated every 3 bits into RGB values ​​(see FIG. 6). Hereinafter, the first RGB value of the second data D2 will be referred to as RGB value Y1 (see FIG. 6). The second RGB value of the second data D2 will be referred to as RGB value Y2. The third RGB value of the second data D2 will be referred to as RGB value Y3. The fourth RGB value of the second data D2 will be referred to as RGB value Y4. The fifth RGB value of the second data D2 will be referred to as RGB value Y5. The sixth RGB value of the second data D2 will be referred to as RGB value Y6.

[0034] Next, the processing unit 201 generates a second video signal Vd2 based on the first video signal Vd1 and the second data D2 (FIG. 4: step S14). Specifically, the processing unit 201 replaces some RGB values ​​of the first video signal Vd1 with RGB values ​​of the second data D2. For example, as shown in FIG. 6, the processing unit 201 specifies one or more regions a1 to a6 in some frames of the first video signal Vd1 (e.g., first frame 300). The regions a1 to a6 each have the same number of pixels, e.g., 4×4 pixels. Hereinafter, the region including the regions a1 to a6 will be referred to as a first region FA.

[0035] Next, the processing unit 201 replaces the RGB values ​​of the regions a1 to a6 with the RGB values ​​of the second data D2. For example, the processing unit 201 replaces the RGB values ​​of the region a3 with the RGB value Y3. In this case, the processing unit 201 replaces the RGB values ​​of the region a3 from (0,0,0) to (0,255,0) (see FIG. 6 ). Similarly, in this embodiment, the processing unit 201 replaces the RGB values ​​of the pixels of the regions a1, a2, a4, a5, and a6 with the RGB values ​​Y1, Y2, Y4, Y5, and Y6, respectively. In other words, the processing unit 201 generates the second video signal Vd2 by replacing the RGB values ​​of the pixels of the regions a1 to a6 in the received first video signal Vd1 with the RGB values ​​of the second data D2. The processing unit 201 may replace the areas a1 to a6 and the areas other than the areas a1 to a6 in the first area FA with a single RGB value (for example, RGB=(0,0,0)).

[0036] When the first video signal Vd1 includes two or more frames, the processing unit 201 replaces the RGB values ​​in each of the two or more frames. For example, as shown in Fig. 3, when the first video signal Vd1 includes a first frame 300 and a second frame 301, the processing unit 201 replaces the RGB values ​​of pixels in regions a1 to a6 of the first frame 300 with the RGB values ​​of the second data D2, and also replaces the RGB values ​​of pixels in regions a1 to a6 of the second frame 301 with the RGB values ​​of the second data D2, thereby generating the second video signal Vd2.

[0037] After generating the second video signal Vd2, in this embodiment, the processing unit 201 converts the format of the second video signal Vd2 (FIG. 4: step S15). The processing unit 201 converts the second video signal Vd2 into a moving image format such as MPEG4. The processing unit 201 outputs the second video signal Vd2 whose format has been converted to the communication interface 202 (see FIG. 5). Then, as shown in FIG. 5, the communication interface 202 outputs the second video signal Vd2 whose format has been converted to the server 40 (FIG. 4: step S16).

[0038] By carrying out the processes from step S11 to step S16, the execution of a series of processes in the video signal processing device 20 is completed (FIG. 4: END).

[0039] The above-described process is merely an example. Therefore, the video signal processing device 20 does not necessarily need to generate the second video signal Vd2 through the above-described process. For example, the processing unit 201 may compress the second video signal Vd2 and output it to the server 40. Furthermore, the processing unit 201 may compress the first data D1 and then convert the compressed first data D1 into a bit string.

[0040] (An example of decoding processing of the second video signal Vd2) The decoding process of the second video signal Vd2 in the terminal 30 will be described below with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the decoding process of the second video signal Vd2.

[0041] First, the terminal 30 receives the second video signal Vd2 (FIG. 7: step S21). Specifically, as shown in FIG.

[0042] Next, the terminal 30 decodes the second video signal Vd2. Specifically, the second video signal Vd2, which has been converted into a moving image format such as MPEG4, is decoded into a data string from which pixel data can be extracted. After that, the terminal 30 extracts the RGB values ​​of some pixels of the second video signal Vd2 and converts them into a bit string (FIG. 7: step S22). The terminal 30 converts each of the RGB values ​​(RGB values ​​Y1 to Y6) of the pixels in areas a1 to a6 into a bit string separated into 3 bits. For example, the terminal 30 converts RGB=(0,255,0) into a bit string of "010".

[0043] Next, the terminal 30 converts the first data D1 into a byte string based on the bit string obtained by the conversion (FIG. 7: step S23). The terminal 30 outputs the first data D1 converted into a byte string (FIG. 7: step S24).

[0044] By executing the processes from step S21 to step S24, the decryption process in the terminal 30 is completed (FIG. 7: END).

[0045] The terminal 30, for example, reads the first data D1 output by the decryption. At this time, the terminal 30 executes processing based on the first data D1. For example, if the decrypted first data D1 includes lighting data, the terminal 30 controls the lighting based on the lighting data. In this case, the terminal 30 functions, for example, as a lighting control device that controls the lighting. Note that the terminal 30 does not necessarily have to read the first data D1. For example, if the terminal 30 is a PC or the like, the terminal 30 may output the first data D1 to a lighting control device.

[0046] (An example of a method for calculating the bit value in step S22) The method for calculating the bit value in step S22 will be described below with reference to Fig. 8. Fig. 8 is a diagram showing an area a1 containing 4x4 pixels.

[0047] 8, area a1 includes 16 pixels from pixel (0,0) to pixel (3,3). In FIG. 8, pixels (0,n), (1,n), (2,n), and (3,n) are arranged in this order along the positive direction of the X axis (n is any number from 1 to 3). Pixels (m,0), (m,1), (m,2), and (m,3) are arranged in this order along the negative direction of the Y axis (m is any number from 1 to 3). In this case, for example, terminal 30 calculates the bit string using (Method 1), (Method 2), (Method 3), or (Method 4) shown below.

[0048] (Method 1) Of the pixels in area a1, the RGB value of the first pixel is extracted and converted into a bit string. For example, the first pixel in area a1 is pixel (0,0). Therefore, terminal 30 extracts the RGB value of pixel (0,0) and converts it into a bit string. For example, if the RGB value of pixel (0,0) in area a1 is RGB=(0,255,0), terminal 30 converts area a1 into a bit string of "010".

[0049] (Method 2) The RGB values ​​of pixels located near the center of area a1 are extracted and converted into a bit string. For example, in the example shown in FIG. 8, pixels (1,1), (2,1), (1,2), and (2,2) are located near the center of area a1. Therefore, terminal 30 converts the RGB value of any of pixels (1,1), (2,1), (1,2), and (2,2) into a bit string. For example, terminal 30 extracts the RGB value of pixel (1,1) and converts it into a bit string.

[0050] (Method 3) The RGB values ​​of pixels located near the center of area a1 are extracted and averaged, and the averaged RGB value is converted into a bit string. For example, if the RGB of pixel (1,1) is (255,0,253), the RGB of pixel (2,1) is (252,0,252), the RGB of pixel (1,2) is (252,0,252), and the RGB of pixel (2,2) is (253,0,255), the average RGB is RGB=(253,0,253). In this case, terminal 30 converts, for example, RGB values ​​of 128 or greater to bits:1 and RGB values ​​of 127 or less to bits:0. Therefore, terminal 30 converts RGB=(253,0,253) into a bit string of "101."

[0051] (Method 4) All RGB values ​​of 4x4 pixels are extracted and averaged. Note that the method for averaging the RGB values ​​of multiple pixels is the same as (Method 3), so explanation is omitted here.

[0052] When the RGB values ​​of multiple pixels are averaged, as in (Method 3) or (Method 4), there is a high possibility that the image can be restored correctly even if the RGB values ​​of some pixels have changed due to noise that occurred during compression.

[0053] The terminal 30 may also include a GPU (Graphics Processing Unit). In this case, the GPU may perform the calculation process of the bit value described above. The GPU has a high speed for calculations related to image processing. Therefore, when the terminal 30 reads the second video signal Vd2, delays in the decoding process are unlikely to occur. This makes it possible to deliver video that requires real-time performance without delays.

[0054] (Effects of the first embodiment) The video signal processing device 20 can distribute both a video signal and data other than the video signal as a single video signal. More specifically, the processing unit 201 generates a second video signal Vd2 by replacing the RGB values ​​of pixels in areas a1 to a6 of the received first video signal Vd1 with RGB values ​​based on the first data D1. This allows the video signal processing device 20 to generate a second video signal Vd2 in which the first data D1 (data other than the video) is embedded within the first video signal Vd1. Existing video distribution platforms can distribute the second video signal Vd2, which is a video signal. Then, a terminal 30 that receives the distributed video signal can decode the first data D1 and perform predetermined processing based on the first data D1. For example, if the first data D1 includes lighting data, the terminal 30 can control the lighting based on the lighting data. This allows the terminal 30 to control the lighting of a public viewing venue in the same way as the lighting of a live venue from which the video is distributed. As a result, users can watch videos of live performances under controlled lighting similar to that of the live performance venue. In this way, distributors can distribute both video signals and data other than video signals using existing video distribution platforms.

[0055] The terminal 30 can synchronize the playback process of the video signal with a process other than the playback of the video signal. Specifically, the second video signal Vd2 contains data (i.e., first data D1) within one video frame that is played back in synchronization with that frame. Therefore, when the terminal 30 reads each frame of the second video signal Vd2, it simultaneously reads the first data D1 embedded in each frame. The terminal 30 then executes the process related to the first data D1 at the same timing as each frame of the video played back by the second video signal Vd2. This allows the terminal 30 to synchronize the playback process of the video signal with a process related to the video signal (for example, lighting control).

[0056] In particular, if the first data D1 includes lighting data, the video displayed based on the second video signal Vd2 is synchronized with the lighting operation (e.g., lighting, blinking, brightness adjustment, etc.). Therefore, for example, the distributor prepares a video of a live performance scene as the first video signal Vd1. Similarly, the distributor prepares the first data D1 including lighting data. Then, the distributor uses the video signal processing device 20 to embed the lighting data (first data) into the first video signal Vd1 to generate the second video signal Vd2. As a result, when the terminal 30 reads the second video signal Vd2, the terminal 30 controls the lighting in accordance with the live performance. Therefore, even if the viewer is located outside the live venue, the viewer can experience the same sense of realism as if they were watching a live performance at the live venue.

[0057] When data is embedded in a video signal (image), for example, there is a method of embedding data in the image by using a digital watermark. In the case of a digital watermark, in addition to the process of embedding data in the video signal, additional processes such as concealing the embedded data are performed. Therefore, when reading data embedded in a video signal by a digital watermark, the terminal needs to execute an algorithm for reading the embedded data and an algorithm for analyzing the concealed information (hereinafter referred to as algorithm V1).

[0058] On the other hand, in this embodiment, the processing unit 201 generates the second video signal Vd2 by replacing the RGB values ​​of pixels in areas a1 to a6 of the first video signal Vd1 with the RGB values ​​of the second data D2. In this case, the terminal 30 reads bit values ​​based on the RGB values ​​during decoding. Here, the algorithm for converting the RGB values ​​into bit values ​​is less complex than the algorithm V1. Therefore, the calculation load incurred in the decoding process of the second data D2 is lower than that of the algorithm V1. As a result, delays are less likely to occur in the decoding process when the terminal 30 is reading the second video signal Vd2. Therefore, in the case of video distribution that requires real-time performance, the video signal processing method of this embodiment can perform distribution and playback processes with lower delays than existing digital watermarking and the like.

[0059] (Modification 1 of the video signal processing device 20) The video signal processing device 20a according to Modification 1 will be described below. A processing unit 201a (not shown) of the video signal processing device 20a compresses the second video signal Vd2 in units of a certain compression block. Here, the number of pixels in the regions a1 to a6 is the same as the number of pixels in the compression block unit of the second video signal Vd2. For example, if the compression block unit of the processing unit 201a is 8 pixels, the processing unit 201a sets the number of pixels in each of the regions a1 to a6 to 8 pixels. Then, the processing unit 201a compresses the second video signal Vd2 in units of 8-pixel compression blocks.

[0060] (Effects of Modification 1) According to the video signal processing device 20, the second video signal Vd2 is less susceptible to compression. If the number of pixels in a compressed block unit differs from the number of pixels in each region to be replaced, for example, the processing unit may combine and compress regions a1 and a2. In this case, the RGB value information of the pixels in region a1 and the RGB value information of the pixels in region a2 are combined, resulting in the loss of information about each region. Therefore, the RGB values ​​of the pixels in regions a1 and a2 may not return to their pre-compression RGB values ​​upon decoding. Meanwhile, in this modification, the number of pixels in regions a1 to a6 is the same as the number of pixels in a compressed block unit of the second video signal Vd2. For example, if the compressed block unit of the second video signal Vd2 is eight pixels, the RGB values ​​of regions a1 to a6 are eight pixels. In this case, for example, the processing unit 201a does not combine and compress regions a1 and a2. Therefore, the RGB value information of the pixels in the areas a1 and a2 is not lost, and the RGB values ​​of the pixels in the areas a1 and a2 are restored to the RGB values ​​before decoding during decoding.

[0061] (Modification 2 of the video signal processing device 20) A video signal processing device 20b according to a second modification of the video signal processing device 20 will be described below with reference to Fig. 3. The video signal processing device 20b compresses the second video signal Vd2 independently for each frame.

[0062] In this modification, a processing unit 201b (not shown) of the video signal processing device 20b replaces the RGB values ​​of regions a1 to a6 of each of two or more frames with the RGB values ​​of the second data D2. The processing unit 201b then independently compresses each frame included in the second video signal Vd2. In other words, the first frame 300 shown in FIG. 3 is intra-frame compressed, and the second frame 301 shown in FIG. 3 is also intra-frame compressed. For example, the processing unit 201b replaces the RGB values ​​of regions a1 to a6 of each of the first frame 300 and the second frame 301 with the RGB values ​​of the second data D2. The processing unit 201b then independently compresses the first frame 300 and the second frame 301. The communication interface 202 outputs the compressed second video signal Vd2.

[0063] (Effects of Modification 2) In this modification, the RGB values ​​of the regions a1 to a6 of each of two or more frames are replaced independently. Inter-frame compression is a method for compressing the second video signal Vd2. Inter-frame compression is a compression method that records only the differences in data between adjacent frames. In other words, inter-frame compression extracts and compresses only the portions where the RGB values ​​differ between multiple frames. In other words, when the second video signal Vd2 is compressed using inter-frame compression, if data with the same RGB values ​​is embedded in the same pixel, the embedded data may be lost. This may result in the RGB values ​​of the pixel not returning to their pre-encoding state during decoding.

[0064] On the other hand, the processing unit 201b independently compresses each frame included in the second video signal Vd2. In this case, unlike inter-frame compression, the data embedded in the second video signal Vd2 (data embedded by replacing RGB values) is less likely to be lost. Therefore, the terminal 30 is more likely to be able to correctly decode the first data D1.

[0065] (Variation 3 of the video signal processing device 20) A video signal processing device 20c according to Modification 3 will be described below with reference to the drawings. FIG. 9 is a diagram showing an example of conversion of first data D1 including identifier data ID. Note that area a6 is omitted in FIG. 9. The video signal processing device 20c differs from the video signal processing device 20 in that it generates a second video signal Vd2 based on first data D1 including identifier data ID. This will be described in detail below.

[0066] The processing unit 201c (not shown) of the video signal processing device 20c generates the first data D1 by, for example, adding identifier data ID to the illumination data CD (see FIG. 9). The identifier data ID is data for determining whether the first data D1 is embedded in the second video signal Vd2 (whether the second video signal Vd2 includes second data). That is, the first data D1 includes identifier data ID for identifying that the first data D1 is embedded. If the first data D1 includes the identifier data ID, it can be determined that the second video signal Vd2 has been generated by the video processing method of this embodiment. In this modification, the first data D1 is arranged in the order of the identifier data ID and illumination data CD from the most significant bit (see FIG. 9). In the example shown in FIG. 9, the identifier data ID is a byte value of "0x55."

[0067] The processing unit 201c converts the first data D1 including the identifier data into second data D2. Note that the processing in the video signal processing device 20c after the processing unit 201c converts the first data D1 into the second data D2 is the same as that in the video signal processing device 20, and therefore a description thereof will be omitted.

[0068] (An example of the decoding process in Modification 3) An example of the decoding process executed by the terminal 30 in the third modification will be described below with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the decoding process executed by the terminal 30 in the third modification.

[0069] In this modification, after step S21, the terminal 30 determines whether or not the second video signal Vd2 includes identifier data ID (FIG. 10: step S32). For example, in the example shown in FIG. 9, the identifier data ID replaced with RGB values ​​is embedded in each of the areas a1, a2, and a3. Therefore, the terminal 30 converts the RGB values ​​of each of the areas a1, a2, and a3 into bit values. Then, the terminal 30 extracts the first 8 bits of the converted bit values. This allows the terminal 30 to determine whether or not the second video signal Vd2 includes identifier data ID. That is, the terminal 30 can determine whether or not the second video signal Vd2 includes identifier data ID by extracting the RGB values ​​of at least three areas included in the first area FA.

[0070] If the terminal 30 determines that the second video signal Vd2 includes the identifier data ID (FIG. 10: Step S32 Yes), the terminal 30 decodes the first data D1 based on the RGB values ​​of the pixels in the areas a1 to a6 in the second video signal Vd2 (FIG. 10: Step S33). After step S33, the terminal 30 executes step S24.

[0071] If the terminal 30 determines that the second video signal Vd2 does not include the identifier data ID (FIG. 10: Step S32 No), the terminal 30 does not decode the first data D1 (FIG. 10: Step S3 4 ).

[0072] (Effects of Modification 3) The video signal processing device 20c first decodes the data block in which the identifier data ID is stored. If the identifier data ID is not stored in the second video signal Vd2, the terminal 30 does not decode the first data D1, and therefore does not perform unnecessary decoding. More specifically, if the terminal 30 determines that the identifier data ID is included in the second video signal Vd2, the terminal 30 decodes the first data D1 based on the RGB values ​​of the pixels in areas a1 to a6 in the second video signal Vd2. Therefore, if decoding of the first data D1 is unnecessary, the terminal 30 can simply reproduce the input video signal without executing the process of decoding the first data D1.

[0073] (Fourth Modification of the Video Signal Processing Device 20) A video signal processing device 20d according to Modification 4 will be described below with reference to Fig. 11. Fig. 11 is a diagram showing an example of processing in the video signal processing device 20d. The video signal processing device 20a differs from the video signal processing device 20 in that the video signal processing device 20a expands the size of the video displayed based on the first video signal Vd1.

[0074] 3, the direction in which the regions a1 to a6 are aligned in the first frame 300 in Fig. 11 is defined as the X-axis direction. Also, the direction perpendicular to the X-axis direction in the first frame 300 is defined as the Y-axis direction.

[0075] The video signal processing device 20d includes a processing unit 201d (not shown) instead of the processing unit 201. The processing unit 201d expands the number of pixels in the first video signal Vd1. For example, as shown in FIG. 11, the processing unit 201d expands the number of pixels in the Y-axis direction of the first frame 300. For example, the processing unit 201d expands the number of pixels in the first video signal Vd1 from 1280 × 720 to 1280 × 724. As a result, as shown in FIG. 11, the first video signal Vd1 has an area AA before expansion and an expanded area EA. The processing unit 201d designates the expanded area EA in the first video signal Vd1 as areas a1 to a6. The processing unit 201d then replaces the RGB values ​​of the pixels in the area EA designated as areas a1 to a6 with the RGB values ​​of the second data D2. For example, if the processing unit 201d expands the number of pixels of the first video signal Vd1 from 1280×720 to 1280×724, the processing unit 201d designates a total of 320 areas, each consisting of 4×4 pixels, from the expanded 1280×4 pixels as replacement areas.

[0076] In the example shown above, the RGB values ​​of the pixels in the expanded 1280x4 area are replaced. In this case, the processing unit 201d displays the original video signal in the area AA without replacing the RGB values ​​of the pixels of the original video signal (first video signal Vd1 before conversion). That is, it is not necessary to replace the RGB values ​​of some pixels of the original video signal (first video signal Vd1 before conversion). In other words, by expanding the pixels, the original image (video based on the first video signal) can be maintained.

[0077] (An example of the decoding process in Modification 4) An example of the decoding process executed by the terminal 30 in the fourth modification will be described below with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the decoding process executed by the terminal 30 in the fourth modification.

[0078] In this modification, the terminal device 30 determines whether to decode the second video signal Vd2 by referring to the number of pixels in the second video signal Vd2. Specifically, after step S21, the terminal device 30 determines whether the number of pixels in the second video signal Vd2 is a specific number of pixels (FIG. 12: step S42). If the terminal device 30 determines that the number of pixels in the second video signal Vd2 is a specific number of pixels (FIG. 12: step S42: Yes), the terminal device 30 decodes the first data D1 based on the RGB values ​​of areas a1 to a6 in the second video signal Vd2 (FIG. 12: step S43). For example, if the video signal processing device 20a is set to expand the number of pixels in the first video signal Vd1 to 1280 × 724, the terminal device 30 determines whether the number of pixels in the second video signal Vd2 is 1280 × 724. If the terminal 30 determines that the number of pixels in the second video signal Vd2 is 1280 × 724, the terminal 30 decodes the first data D1 based on the RGB values ​​of the areas a1 to a6 in the second video signal Vd2, and outputs the decoded first data D1 (FIG. 12: Step S24).

[0079] If the terminal 30 determines that the number of pixels in the second video signal Vd2 is not a specific number of pixels (FIG. 12: Step S42 No), the terminal 30 does not decode the first data D1 based on the RGB values ​​of the areas a1 to a6 in the second video signal Vd2 (FIG. 12: Step S44).

[0080] (Effects of Modification 4 on Decryption Processing) The video signal processing device 20d can prevent the terminal 30 from performing unnecessary decoding processing. Specifically, the terminal 30 decodes the first data D1 only when the number of pixels in the second video signal Vd2 is a specific number of pixels. Therefore, if the number of pixels in the second video signal Vd2 is not the specific number of pixels, the terminal 30 does not decode the first data D1. Therefore, the terminal 30 does not perform unnecessary processing.

[0081] (Fifth Modification of the Video Signal Processing Device 20) A video signal processing device 20e according to Modification 5 will be described below with reference to Fig. 13. Fig. 13 is a diagram showing an example of processing in the video signal processing device 20e. The video signal processing device 20e differs from the video signal processing device 20 in that it converts the RGB values ​​of pixels in the second area SA in addition to areas a1 to a6.

[0082] A processing unit 201e (not shown) of the video signal processing device 20e specifies a second area SA as shown in Fig. 13. The second area SA does not overlap with areas a1 to a6. A part of the second area SA borders a part of areas a1 to a6. The processing unit 201f sets the RGB values ​​of the pixels in the second area SA to a single RGB value (for example, RGB = (0,0,0)).

[0083] Furthermore, the processing unit 201f designates an area other than the areas a1 to a6 and the second area SA as the third area TA. In this case, the second area SA exists between the areas a1 to a6 and the third area TA. A video is reproduced in the third area TA. That is, the RGB values ​​of the third area TA are likely to change from frame to frame. Therefore, if the areas a1 to a6 and the third area TA are adjacent when the second video signal Vd2 is generated, the RGB values ​​of the third area TA may become noise and affect the RGB values ​​of the areas a1 to a6. On the other hand, in this modification, the areas a1 to a6 are adjacent to the second area SA. Because the RGB values ​​of the pixels in the second area SA are uniform, the RGB values ​​of the second area SA are unlikely to become noise and affect the RGB values ​​of the areas a1 to a6.

[0084] (Application example 1 of terminal 30 processing) Hereinafter, an application example 1 of the processing of the terminal 30 will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of the processing in the terminal 30.

[0085] In this application example, the terminal 30 receives a second video signal Vd2. The terminal 30 then generates a third video signal Vd3 by removing areas a1 to a6 from the second video signal Vd2. In the example shown in FIG. 14, the processing unit 201b removes a first area FA that includes areas a1 to a6. For example, if the number of pixels in the second video signal Vd2 before removal is 1280 × 720 and the number of pixels in areas a1 to a6 is 1280 × 4, the processing unit 201b changes the number of pixels in the second video signal Vd2 to 1280 × 716. In other words, the processing unit 201b changes the resolution of the second video signal Vd2.

[0086] As a result, the areas a1 to a6 are not displayed in the video displayed based on the third video signal Vd3. That is, the terminal 30 displays the video according to the video signal before the RGB values ​​were replaced (the original video signal). Therefore, the user can watch the distributed video without feeling any discomfort.

[0087] In this application example, the processing unit 201b does not necessarily have to remove the regions a1 to a6 by changing the resolution of the second video signal Vd2. For example, the processing unit 201b may change the RGB values ​​of the region including the regions a1 to a6 to a single RGB value. For example, if the number of pixels in the regions a1 to a6 is 1280 × 4, the processing unit 201b changes the RGB values ​​of each of the 1280 × 4 pixels to, for example, RGB values ​​= (0, 0, 0). In this case, the resolution of the second video signal Vd2 remains 1280 × 720.

[0088] (Application Example 1 of Video Signal Processing Devices 20, 20a to 20e) Hereinafter, an application example 1 of the video signal processing devices 20, 20a to 20e will be described with reference to the drawings. FIG. 15 is a diagram showing application example 1 of the video signal processing devices 20, 20a to 20e. In the application example, the video signal processing devices 20, 20a to 20e output a second video signal Vd2 related to VR (Virtual Reality). The terminal 30 plays back VR video based on the second video signal Vd2. In this case, the terminal 30 is, for example, a PC. Then, the terminal 30 displays a virtual 3D space on the display of the PC. Hereinafter, details of the processing related to the application example will be described in detail.

[0089] Terminal 30 generates a virtual space VS based on virtual space generation data (not shown) stored in advance in the terminal 30 (see FIG. 15). Specifically, the virtual space generation data is data that determines the shape (cubic, spherical, etc.) or size (coordinates of the edges of the virtual space VS, etc.) of the virtual space VS. For example, the virtual space generation data includes coordinates of the edges of the virtual space VS in the X-axis direction, the Y-axis direction, and the Z-axis direction. FIG. 15 shows a cubic virtual space VS as an example. The virtual space VS is expressed as an example using coordinates of x=0-1, y=0-1, and z=0-1, with G0 in FIG. 14 as the origin.

[0090] In this application example, the first data D1 includes spatial coordinate data SD of the virtual space VS. The spatial coordinate data SD is data indicating position information of an object OBJ, which is a virtual object placed in the virtual space VS. The spatial coordinate data SD includes, for example, data indicating the position, shape, or size of the object OBJ. The terminal 30 places the object OBJ (e.g., a screen and lighting) in the virtual space VS based on the spatial coordinate data SD. In other words, the first data D1 includes spatial coordinate data SD of the object OBJ displayed in the virtual space VS.

[0091] The terminal 30 obtains the spatial coordinate data SD from the second video signal Vd2 by decoding the second video signal Vd2. The terminal 30 performs display based on the spatial coordinate data SD. For example, the terminal 30 generates a virtual space VS based on virtual space generation data stored in the terminal 30. The terminal 30 reads the spatial coordinate data SD included in the second video signal. The terminal 30 then places an object OBJ in the virtual space VS based on the coordinates of the spatial coordinate data SD. In this way, the terminal 30 displays the virtual space VS based on the second video signal Vd2.

[0092] In this application example, the object OBJ includes, for example, a screen SC that displays the second video signal Vd2 in the virtual space VS. In this case, the first data D1 includes spatial coordinate data SD that indicates the position of the screen SC. The terminal 30 displays the screen SC based on the coordinates of the screen SC, etc. In other words, the terminal 30 displays the screen SC in the virtual space VS based on the spatial coordinate data SD. In this case, the spatial coordinate data SD includes coordinates of diagonal positions of the screen SC, center coordinates of the screen SC, or the size of the screen SC. For example, the spatial coordinate data SD includes coordinates of diagonal positions of the screen SC (lower left LD: x1, y1, z1; upper left LU: x2, y2, z2; upper right RU: x3, y3, z3; lower right RD: x4, y4, z4). The spatial coordinate data SD includes, for example, the following coordinates: lower left LD: (x1, y1, z1) = (0.2, 1.0, 0.4), upper left LU: (x2, y2, z2) = (0.2, 1.0, 0.7), upper right RU: (x3, y3, z3) = (0.8, 1.0, 0.7), lower right RD: (x4, y4, z4) = (0.8, 1.0, 0.4). This allows the terminal 30 to display objects on the screen SC within the virtual space VS. The terminal 30 then displays an image based on the second video signal Vd2 on the screen SC within the virtual space VS.

[0093] The object OBJ also includes information about the lights L1VR and L2VR (see FIG. 15). The information about the lights L1VR and L2VR includes, for example, the position (coordinates) or direction of the lights. The terminal 30 displays the lights L1VR and L2VR in the virtual space VS based on the coordinates or direction of the lights L1VR and L2VR. The terminal 30 controls the lights L1VR and L2VR (e.g., turns them on, off, changes their color, etc.) based on the lighting data embedded in the second video signal Vd2. Therefore, for example, a broadcaster can recreate the lighting of a real-world live venue by setting the positions of the lights L1VR and L2VR based on the position or direction of the lights in the real-world live venue. This allows viewers to not only watch a video of a live performance, but also experience the sense of realism of being in a live music venue and listening to the live performance.

[0094] The information about the lighting L1VR and L2VR may include, for example, the model name of the lighting (model name of the lighting) or information about the lighting's function (monochromatic lighting, color lighting, etc.). In this case, the terminal 30 stores, for example, a 3D model image of the lighting in advance. The 3D model image of the lighting is a model image that reproduces the shape, color, etc. of a lighting in the real world. When the terminal 30 receives information about the model name of the lighting or the lighting's function, the terminal 30 selects a 3D model image of the lighting corresponding to the model name or the lighting's function. The terminal 30 displays the lighting L1VR and L2VR in the virtual space VS based on the selected 3D model image of the lighting. As a result, the terminal 30 displays the lighting L1VR and L2VR that reproduce the shape, etc. of a lighting in the virtual space VS. This allows the broadcaster to further reproduce the lighting, etc. of a live venue.

[0095] For example, as shown in FIG. 15, the terminal 30 displays a bird's-eye view of the virtual space VS. The viewer can view a virtual screen SC present in the virtual space VS via the terminal 30. This allows the viewer to feel as if they are watching a live performance taking place at a live venue from above. For example, the distributor prepares a video of a live performance scene as the first video signal Vd1. In this case, the live performance scene is displayed on the screen SC. Therefore, the viewer can view the live performance taking place within the virtual space VS.

[0096] The spatial coordinate data SD may include the origin of the display in the virtual space VS. The origin of the display in the virtual space VS is, for example, information indicating the viewing position of viewer G in FIG. 15, and is information indicating the origin of the viewer's viewpoint. The spatial coordinate data SD includes, for example, information that the viewing position of viewer G (the origin of viewer G's viewpoint) is (x, y, z) = (0.1, 0.6, 0) (for example, the position of viewer G shown in FIG. 14). In this case, the terminal 30 displays the virtual space VS from a viewpoint looking in a certain direction (for example, the direction of the screen SC) from the viewing position: (x, y, z) = (0.1, 0.6, 0). In other words, the terminal 30 displays the virtual space VS based on the origin of the display. This allows the viewer not only to simply watch a video of a live performance, but also to experience the sense of realism of being in a live music venue and listening to the live performance.

[0097] The terminal 30 may change the viewing position or the viewpoint of the viewer for each frame. For example, in the example shown in FIG. 15, the terminal 30 may switch the viewing position of viewer G from (x, y, z) = (0.1, 0.6, 0) to (x, y, z) = (0.7, 0.6, 0). The terminal 30 may also move the viewing position of viewer G little by little in the positive direction of the X axis for each frame. This allows the terminal 30 to switch images in a way that replicates camerawork in live video, or to move the camera in response to the movement of a subject.

[0098] In this application example, the first data D1 may include, for example, a control program (hereinafter referred to as program P) that executes processing related to the virtual space VS. For example, the video signal processing devices 20, 20a to 20e generate the second video signal Vd2 by embedding the first data D1 including the program P into the first video signal Vd1. When the terminal 30 receives the second video signal Vd2, the terminal 30 reads the program P embedded in the second video signal Vd2. Then, the terminal 30 executes the program P to generate the virtual space VS, an object OBJ, or the like. In this case, the program P is generated by, for example, HTML, JavaScript (registered trademark), or the like.

[0099] Note that the first data D1 does not necessarily have to include the screen SC and the lights L1VR, L2VR as objects OBJ. For example, if the first data D1 includes a program P, the program P may include position information, etc., of the screen SC and the lights L1VR, L2VR (the position information, etc., of the screen SC and the lights L1VR, L2VR may be embedded in the program P). In this case, the terminal 30 executes the program P to read the position information, etc., of the screen SC and the lights L1VR, L2VR included in the program P. Therefore, the broadcaster embeds a lighting model that reproduces the lighting position or shape, etc., of the live venue into the program P. By loading the lighting model embedded in the program P, the terminal 30 displays the lights L1VR, L2VR that reproduce the lighting of the live venue in the virtual space VS.

[0100] (Application Example 2 of Video Signal Processing Devices 20, 20a to 20e) An application example 2 of the video signal processing devices 20, 20a to 20e will be described below. The video signal processing devices 20, 20a to 20e according to application example 2 receive first data D1 including sound control data. The video signal processing devices 20, 20a to 20e generate a second video signal Vd2 based on the first data D1 including the sound control data and output the second video signal Vd2 to the terminal 30. This allows the terminal 30 to control the sound of the video being played back based on the sound control data. The sound control data includes, for example, a volume value, an effect value (e.g., equalizer, delay, etc.), or data related to sound image localization processing. For example, if the sound control data includes a volume value, the terminal 30 increases or decreases the volume of the video being played back according to the volume value.

[0101] (Application Example 3 of Video Signal Processing Devices 20, 20a to 20e) Application example 3 of the video signal processing devices 20, 20a to 20e will be described below. The video signal processing devices 20, 20a to 20e according to application example 3 receive first data D1 including video control data. The video signal processing devices 20, 20a to 20e generate second video signals Vd2 based on the first data D1 including the video control data and output the second video signals Vd2 to the terminal 30. The terminal 30 controls the video to be played based on the video control data. The video control data includes, for example, data related to video division, data related to video effect processing (e.g., screen muting or brightness adjustment), data related to the screen on which the video is displayed, or data related to the position information of the performers. For example, if the video control data includes data related to the brightness of the video, the terminal 30 adjusts the brightness of the video to be played.

[0102] Furthermore, if the video control data includes data related to video division, the terminal 30, for example, divides the video into multiple parts according to the division data. Then, the terminal 30 plays each of the divided videos on a different screen. For example, a distributor prepares multiple cameras (e.g., a first camera and a second camera) at a live venue. Then, for example, in a live performance scene, the distributor uses the first camera to capture the stage of the live venue and the second camera to capture the faces of the performers. At this time, the video signal processing device 20, 20a to 20e receives video signals from each of the first camera and the second camera. Then, the video signal processing device 20, 20a to 20e generates a first video signal Vd1 (hereinafter referred to as a first video signal for division) that embeds both the video signal received from the first camera (hereinafter referred to as the video signal of the first camera) and the video signal received from the second camera (hereinafter referred to as the video signal of the second camera).

[0103] The video signal processing devices 20, 20a to 20e generate second video signals Vd2 based on the first video signals for division. At this time, the video signal processing devices 20, 20a to 20e embed information indicating which of the multiple screens the video signal from the first camera should be displayed on into the second video signal Vd2, and also embed information indicating which of the multiple screens the video signal from the second camera should be displayed on into the second video signal Vd2. For example, the video signal processing devices 20, 20a to 20e embed information indicating that the video signal from the first camera should be displayed on the first screen (hereinafter referred to as first information) and information indicating that the video signal from the second camera should be displayed on the second screen (hereinafter referred to as second information) into the second video signal Vd2. The video signal processing devices 20, 20a to 20e output the second video signal Vd2 to the terminal 30.

[0104] Terminal 30 splits second video signal Vd2 into a video signal from the first camera and a video signal from the second camera. Terminal 30 transmits the video signal from the first camera and the video signal from the second camera to different screens. For example, terminal 30 displays a video of a stage at a live venue (video based on the video signal from the first camera) on the first screen based on the first information, and displays a video of the performers' faces (video based on the video signal from the second camera) on the second screen based on the second information.

[0105] Note that terminal 30 may, for example, switch between the video signal of the first camera and the video signal of the second camera to display the video signal of the first camera and the video signal of the second camera on one screen. In this case, the user of terminal 30 may perform an operation to switch between the video signal of the first camera and the video signal of the second camera via terminal 30.

[0106] The video control data may include data related to camerawork. Examples of the data related to camerawork include the order in which video signals are displayed on the screen. For example, the video control data may include data for displaying, on the first screen, video based on the video signal from the first camera, video based on the video signal from the second camera, and video based on the video signal from the first camera in that order. In this case, the first screen displays, on the first screen, video based on the video signal from the first camera, video based on the video signal from the second camera, and video based on the video signal from the first camera in that order.

[0107] In Application Example 3, the first data D1 may include, for example, performer position data. The terminal 30 may change the position of the screen that plays the video of the performer's face based on the performer position data. For example, the first data D1 may include information such as the direction or distance moved by the performer while performing. In this case, the terminal 30 obtains information on the direction or distance moved by the performer while performing by reading the first data D1. Then, for example, the terminal 30 moves the second screen (the screen that plays the video of the performer's face) in the same direction as the performer's movement. At this time, the terminal 30 changes the amount of movement of the second screen based on the information on the distance moved by the performer. In this application example, the terminal 30 may perform sound image localization processing based on the performer's position data. For example, the terminal 30 may perform sound image localization processing to localize the sound generated by performer C's performance in the direction that performer C (not shown) moved.

[0108] (Application Example 4 of Video Signal Processing Devices 20, 20a to 20e) Hereinafter, an application example 4 of the video signal processing devices 20, 20a to 20e will be described. The video signal processing devices 20, 20a to 20e according to application example 4 embed information indicating the data type of the first data D1 into the second video signal Vd2. The data type is label information for identifying the type of the first data D1. The data type is information indicating, for example, that the first data D1 is audio control data, video control data, or the like. That is, the first data D1 includes a data type, which is label information for identifying the type of the first data D1. The video signal processing devices 20, 20a to 20e embed the data type into the second video signal Vd2. The terminal 30 analyzes the data type embedded in the second video signal Vd2. If the terminal 30 determines, as a result of the analysis, that the first data D1 includes a data type indicating control data executable by the terminal 30, the terminal 30 performs control based on the control data. For example, if an audio control application is installed on the terminal 30, the terminal 30 determines whether the second video signal Vd2 includes the data type of audio control data. If the terminal 30 determines that the second video signal Vd2 includes the data type of audio control data, it controls the audio equipment based on the audio control data.

[0109] (Other embodiments) The video signal processing devices 20, 20a to 20e according to the present invention are not limited to the above and may be modified within the scope of the present invention. Furthermore, the configurations of the video signal processing devices 20, 20a to 20e may be combined in any manner.

[0110] The processing of the video signal processing devices 20, 20a to 20e in the first embodiment is not limited to the example shown in Fig. 4. For example, the processing unit 201 may receive the first data D1 after receiving the first video signal Vd1. Also, for example, the processing unit 201 may receive the first video signal Vd1 after generating the second data D2.

[0111] Note that the video signal processing devices 20, 20a to 20e do not necessarily have to receive the first data D1 and the first video signal Vd1 from a device (hereinafter referred to as device X) different from the video signal processing device 20 (not shown). The video signal processing devices 20, 20a to 20e may, for example, generate the first data D1 or the first video signal Vd1 themselves. In this case, for example, an application program for generating the first data D1 or an application program for generating the first video signal Vd1 is installed in the video signal processing devices 20, 20a to 20e.

[0112] The number of frames included in the first video signal Vd1 is not limited to the example of two shown in FIG.

[0113] The first video signal Vd1 does not necessarily have to include two or more frames. The first video signal Vd1 may include only one frame. In this case, the first video signal Vd1 is a still image such as a photograph.

[0114] Note that by compressing the second video signal Vd2 without increasing the compression ratio (by compressing at high image quality), RGB values ​​are less likely to change at and near the boundaries between the areas a1 to a6 and areas other than the areas a1 to a6, which increases the likelihood that the first data D1 can be correctly decoded.

[0115] The byte value of the first data D1 may be a value other than "0x11" or "0x13".

[0116] The byte value of the identifier data ID may be a value other than "0x55." For example, the video signal processing devices 20, 20a to 20e may set the bit string of the identifier data ID based on a combination of dots that are unlikely to appear in a video signal. For example, if a 4x4 pixel area is defined as one dot, it is unlikely that a dot with RGB=(255,0,0), a dot with RGB=(0,255,0), and a dot with RGB(0,0,255) will be arranged in this order in the video signal. Therefore, the video signal processing devices 20, 20a to 20e set the bit string of the identifier data ID so that the RGB of area a1 is (255,0,0), the RGB of area a2 is (0,255,0), and the RGB of area a3 is (0,0,255). That is, the video signal processing devices 20, 20a to 20e may set the bit string of the identifier data ID to "10001000" (ie, "0x88").

[0117] The number of pixels in each of the regions a1 to a6 does not necessarily have to be 4. For example, the number of pixels in each of the regions a1 to a6 may be 8 pixels or the like.

[0118] It should be noted that the video signal processing devices 20, 20a to 20e do not necessarily have to specify the second area SA.

[0119] The number of pixels in the regions a1 to a6 does not necessarily have to be the same as the number of pixels in the compressed block unit of the second video signal Vd2.

[0120] It should be noted that the video signal processing devices 20, 20a to 20e do not necessarily have to compress each frame included in the second video signal Vd2 independently.

[0121] The first data D1 does not necessarily have to include illumination data.

[0122] The first data D1 does not necessarily have to include the spatial coordinate data SD.

[0123] Note that the object OBJ does not necessarily have to include the screen SC.

[0124] In Application Example 1, the object OBJ does not necessarily have to include information on two lights (lights L1VR and L2VR). For example, the object OBJ may include information on one light, or may include information on three or more lights.

[0125] The first data D1 does not necessarily have to expand the number of pixels of the first video signal Vd1. Furthermore, the video signal processing devices 20, 20a to 20e do not necessarily have to specify the expanded regions in the first video signal Vd1 as the regions a1 to a6.

[0126] It should be noted that the video signal processing devices 20, 20a to 20e do not necessarily have to generate the third video signal Vd3 by removing the regions a1 to a6 from the second video signal Vd2.

[0127] The terminal 30 does not necessarily have to determine whether the second video signal Vd2 includes the identifier data ID. In this case, the terminal 30 may decode the first data D1 even if it has not determined that the second video signal Vd2 includes the identifier data ID.

[0128] The terminal device 30 does not necessarily have to determine whether the number of pixels in the second video signal Vd2 is the specific number of pixels. In this case, the terminal device 30 may decode the first data D1 even if it has not determined that the number of pixels is the specific number of pixels.

[0129] The first data D1 may further include data other than the identifier data ID. The first data D1 may include, for example, data indicating the number of pieces of data or a checksum. The data indicating the number of pieces of data is a byte sequence recording the number of pieces of data in the first data D1. The checksum is data for verifying whether the second video signal Vd2 before being transmitted to the terminal 30 is the same as the second video signal Vd2 after being transmitted to the terminal 30. For example, the video signal processing device 20, 20a to 20e calculates the checksum based on the data sequence of the first data D1 before being transmitted to the terminal 30. Furthermore, the video signal processing device 20, 20a to 20e calculates the checksum based on the data sequence of the first data D1 after being transmitted to the terminal 30. At this time, if the checksum is correct, the terminal 30 may determine that the second video signal Vd2 has been successfully decoded. The first data D1 may include, for example, an error correction code such as a Reed-Solomon code.

[0130] The terminal 30 may determine whether the first data D1 has been successfully decoded based on the identifier data ID. For example, if the terminal 30 can decode a byte value of "0x55", the terminal 30 determines that the first data D1 has been successfully decoded. On the other hand, if the terminal 30 cannot decode the byte value of "0x55", the terminal 30 determines that the first data D1 cannot be decoded.

[0131] The first data D1 may include data other than the sound control data described in Application Example 2 or the video control data described in Application Example 3. For example, the first data D1 may include control data for a PC or the like (e.g., control data for turning a power supply on and off), or control data for a home appliance controller (e.g., control data for opening and closing curtains, control data for turning indoor lights on and off, or control data for turning an electric fan on and off).

[0132] In the fourth modification, when the terminal 30 receives the second video signal Vd2 including the expanded region EA, the terminal 30 may remove the region EA. For example, if the number of pixels in the second video signal Vd2 is expanded from 1280 × 720 to 1280 × 724 by the region EA, the terminal 30 may change the number of pixels in the second video signal Vd2 to 1280 × 720. That is, the terminal 30 may change the resolution of the second video signal Vd2. Alternatively, the terminal 30 may change the RGB values ​​of the expanded region EA to a single RGB value. When the terminal 30 removes the expanded region EA from the second video signal Vd2 by changing the resolution of the second video signal Vd2, the terminal 30 displays the original video signal (the first video signal Vd1 before conversion). Therefore, when the terminal 30 changes the resolution of the second video signal Vd2, the user can watch the distributed video without feeling uncomfortable, compared to when the terminal 30 changes the RGB values ​​of the region EA to a single RGB value.

[0133] The first data D1 may include both the identifier data ID and the data type. In this case, the identifier data ID and the data type are arranged in the first data D1 in the order of the identifier data ID and the data type from the most significant bit. The video signal processing devices 20, 20a to 20e embed the first data D1 including the identifier data ID and the data type into the first video signal Vd1 to generate the second video signal Vd2. When the terminal 30 determines that the identifier data ID is included in the second video signal Vd2, it decodes the data type. [Explanation of symbols]

[0134] 20, 20a to 20e...Video signal processing device 200...Indicator 201...Processing section 202...Communication interface 203...User Interface 204...Flash memory 205...RAM D1...First data D2...Second data Vd1...1st video signal a1~a6…field FA...First Domain Vd2…2nd video signal

Claims

1. Accept the first data, generating second data by converting the received first data into RGB values; receiving a first video signal including RGB values ​​for each pixel; designating an edge of a display area of ​​an image based on the received first video signal as a first area; generating a second video signal by replacing the RGB values ​​of the pixels in the first region of the first video signal with the RGB values ​​of the second data; outputting the second video signal; Video signal processing method.

2. Accepting the first data, generating second data by converting the received first data into RGB values; receiving a first video signal including RGB values ​​for each pixel; Expanding a display area of ​​an image based on the received first video signal; designating the expanded display area as a first area; generating a second video signal by replacing the RGB values ​​of the pixels in the first region of the first video signal with the RGB values ​​of the second data; outputting the second video signal; Video signal processing method.

3. The second video signal is compressed and output in blocks each consisting of a predetermined number of pixels, the first region includes one or more regions each having the same number of pixels as the predetermined number of pixels; replacing the RGB values ​​of the pixels of the one or more regions with the RGB values ​​of the second data; 3. The video signal processing method according to claim 1.

4. the first video signal includes two or more frames; replacing the RGB values ​​of the first region of each of the two or more frames with the RGB values ​​of the second data; independently compressing each frame included in the second video signal; outputting the second video signal after compression; 4. The video signal processing method according to claim 1.

5. the first data includes lighting data for controlling lighting; the lighting data is data for controlling the operation of the lighting in synchronization with the image of the first video signal; 5. A video signal processing method according to claim 1.

6. the first data includes spatial coordinate data of a screen as an object arranged in a virtual space displayed on a terminal that receives the second video signal; In the terminal, the screen is arranged and displayed in the virtual space based on the spatial coordinate data; In the terminal, an image based on the second video signal is displayed on the screen displayed in the virtual space.

6. A video signal processing method according to claim 1.

7. the first data includes an origin of a display in a virtual space displayed on a terminal that receives the second video signal; displaying the virtual space based on the origin on the terminal; 7. The video signal processing method according to claim 1.

8. The first data includes identifier data for identifying that the first data is embedded.

8. The video signal processing method according to claim 1.

9. receiving the second video signal; determining whether the second video signal includes the identifier data; when it is determined that the identifier data is included in the second video signal, decoding the first data based on RGB values ​​of pixels in the first region in the second video signal; 9. The video signal processing method according to claim 8.

10. receiving the second video signal; outputting a third video signal obtained by removing the first region from the second video signal; 10. The video signal processing method according to claim 1.

11. receiving the second video signal; determining whether the number of pixels of the second video signal is a specific number of pixels; When it is determined that the number of pixels is the specific number of pixels, decoding the first data based on the RGB values ​​of the first region in the second video signal.

11. A video signal processing method according to claim 1.

12. a processing unit that receives first data and a first video signal including RGB values ​​for each pixel, and generates second data by converting the received first data into RGB values, wherein the processing unit designates an edge of a display area of ​​an image based on the received first video signal as a first area, and generates the second video signal by replacing the RGB values ​​of pixels in the first area of ​​the first video signal with the RGB values ​​of the second data; an output unit that outputs the second video signal; Equipped with Video signal processing device.

13. A processing unit that receives first data and a first video signal containing RGB values ​​for each pixel, and generates second data by converting the received first data into RGB values, wherein the processing unit expands a display area of ​​an image based on the received first video signal, designates the expanded display area as a first area, and generates a second video signal by replacing the RGB values ​​of the pixels of the first area of ​​the first video signal with the RGB values ​​of the second data; an output unit that outputs the second video signal; Equipped with Video signal processing device.

14. The processing unit compresses the second video signal in blocks each consisting of a predetermined number of pixels and outputs the compressed signal; the first region includes one or more regions each having the same number of pixels as the predetermined number of pixels; the processing unit replaces RGB values ​​of pixels in the one or more regions with RGB values ​​of the second data; 14. A video signal processing device according to claim 12 or 13.

15. the first video signal includes two or more frames; the processing unit replaces the RGB values ​​of the first region of each of the two or more frames with the RGB values ​​of the second data; the processing unit independently compresses each frame included in the second video signal; the output unit outputs the compressed second video signal.

15. A video signal processing device according to claim 12.

16. the first data includes lighting data for controlling lighting; the lighting data is data for controlling the operation of the lighting in synchronization with the image of the first video signal; 16. A video signal processing device according to claim 12.

17. The first data includes identifier data for identifying that the first data is embedded.

17. A video signal processing device according to claim 12.

18. the video signal processing device is communicably connected to a terminal different from the video signal processing device; 18. A video signal processing device according to claim 12.

19. the terminal receives the second video signal; the terminal generates a third video signal by removing the first region from the second video signal; The terminal outputs the third video signal.

19. A video signal processing device according to claim 18.

20. The first data includes spatial coordinate data of a screen as an object placed in a virtual space displayed on the terminal that receives the second video signal; In the terminal, the screen is arranged and displayed in the virtual space based on the spatial coordinate data; In the terminal, an image based on the second video signal is displayed on the screen displayed in the virtual space.

20. A video signal processing device according to claim 18 or 19.

21. The first data includes an origin of a display in a virtual space displayed on the terminal that receives the second video signal; the terminal displays the virtual space based on the origin.

21. A video signal processing device according to claim 18.

22. the first data includes identifier data for identifying a type of the first data; The terminal receives the second video signal, The terminal determines whether the second video signal includes the identifier data; If the terminal determines that the identifier data is included in the second video signal, the terminal decodes the first data based on RGB values ​​of pixels in the first region in the second video signal.

22. A video signal processing device according to claim 18.

23. The terminal receives the second video signal, The terminal determines whether the number of pixels of the second video signal is a specific number of pixels; When the terminal determines that the number of pixels is the specific number of pixels, the terminal decodes the first data based on the RGB values ​​of the first region in the second video signal.

23. A video signal processing device according to any one of claims 18 to 22.

Images