Video encoding device and program

The video encoding device links video and audio objects to guide viewer attention, adjusting encoding quality based on audio metadata, addressing inefficiencies in conventional encoding methods and enhancing the viewing experience for ultra-high definition, large-screen, and wide-viewing-angle videos.

JP7587391B2Active Publication Date: 2024-11-20NIPPON HOSO KYOKAI

Patent Information

Application Number
JP2020176680
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-10-21
Publication Date
2024-11-20
Estimated Expiration
2040-10-21

AI Technical Summary

Technical Problem

Conventional video encoding technologies fail to efficiently encode ultra-high definition, large-screen, and wide-viewing-angle videos by uniformly encoding entire frames, neglecting the viewer's focus on specific regions of interest, and lack integration with audio metadata to guide the viewing experience.

Method used

A video encoding device that links video and audio objects to determine a guided gaze area based on audio metadata, adjusting encoding quality to maintain high quality in regions of interest, using video and audio metadata to guide the viewing area and control encoding parameters.

Benefits of technology

This approach compresses data effectively without reducing subjective quality, enhancing the viewing experience by aligning encoding with viewer attention and maintaining high image quality in regions of interest.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007587391000001
    Figure 0007587391000001
  • Figure 0007587391000002
    Figure 0007587391000002
  • Figure 0007587391000003
    Figure 0007587391000003
Patent Text Reader

Abstract

To provide a video encoder and a program, guiding the gaze area of a video depending on video and audio and encoding video while keeping the quality of video contents at a high level.SOLUTION: A video encoder 1 of the present invention includes: a video object position / shape extraction section 2 which inputs a video before encoding processing and extracts the position coordinates and shape coordinates of a video object by video metadata accompanying the video or video analysis; an audio position generation section 3 which inputs audio metadata and audio accompanying a video and associates the position coordinates of the video object with information of the loudness of the sound of the video object, the video object being linked with an audio object; a guided gaze area generation section 4 which decides a video object to be guided and gazed among video objects linked as a sounder based on a predetermined criteria, and determines a guided gaze area: an encoding control information generation section 5 to control a code amount so as to improve the coded image quality in the guided gaze area over other areas; and a video encoding section 6 to encode the video based on the control of the code amount.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a video encoding device and a program for encoding and transmitting video having characteristics of ultra-high definition, a large screen, and a wide viewing angle. [Background technology]

[0002] Conventionally, there has been known a video-related content generating device that analyzes an input video signal, detects the position coordinates of a video object related to a video scene, and generates video-related content by associating it with character string data and audio data related to the content of the video that has been prepared in advance (see, for example, Patent Document 1). Therefore, before encoding the video signal, it is possible to obtain the video signal that is the original signal, video metadata indicating the position coordinates of the video object, and audio data related to the content of the video.

[0003] Also, various techniques are known for detecting the position coordinates and shape coordinates of a video object from a video frame of an input video signal (see, for example, Patent Document 2).

[0004] In recent years, immersive media (highly realistic media) is expected to be the future of video media. In addition to immersive media, "Diverse Vision" is being considered as a service that allows anyone to watch and experience various contents using their preferred devices regardless of time or place (see, for example, Non-Patent Document 1).

[0005] These future video systems will provide a highly realistic video experience, and the images used in these video systems will have characteristics such as ultra-high definition, large screen, wide viewing angle, etc. The amount of data for such images will be extremely huge, and in order to realize services that encode and transmit images with the characteristics of ultra-high definition, large screen, and wide viewing angle, an extremely efficient and high-performance video encoding device is required.

[0006] In addition, audio coexists in conjunction with video in all video content, not just in immersive media. There are various methods for expressing audio data depending on the system, but for future advanced content, systems such as object-based audio systems are being considered in which speakers and sound sources are defined as objects, and various information associated with them, such as playback position information of audio objects, is added as metadata (for example, see Non-Patent Document 2).

[0007] For example, a real-time transmission device for audio metadata adds audio metadata to an audio signal (see, for example, Non-Patent Document 3). Incidentally, ITU-R specifies the Audio Definition Model (ADM) as an international standard for audio metadata, and the ADM in this audio metadata consists of a content section that describes the program contents, and a format section that describes speaker placement, object playback positions, etc. Metadata that changes over time, such as the playback positions of objects, is called dynamic metadata, and can be handled separately from static metadata such as the content section that does not change throughout the program. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2004-15523 A [Patent Document 2] JP 2001-307091 A [Non-patent literature]

[0009] [Non-Patent Document 1] “Diverse Vision: Media Technology around 2030-2040”, [online], NHK Science and Technology Research Laboratories, Open House 2019, held on May 30, 2019, [Retrieved September 18, 2020], Internet〈https: / / www.nhk.or.jp / strl / open2019 / tenji / e1.html〉 [Non-Patent Document 2] “Object-based acoustics”, [online], NHK Science and Technology Research Laboratories, Giken Dayori, January 2019 issue, [Retrieved September 18, 2020], Internet〈https: / / www.nhk.or.jp / strl / publica / giken_dayori / 166 / 5.html〉 [Non-Patent Document 3] Kubo and Oide, "Development of a real-time transmission device for audio metadata in object-based audio", 2019 ITE Winter Conference, 22A-3, presented on December 13, 2019 Summary of the Invention [Problem to be solved by the invention]

[0010] As mentioned above, in order to provide services that encode and transmit super-high definition, large screen, and wide viewing angle video using a transmission line with limited transmission capacity, such as broadcasting and distribution, video encoding that compresses data according to the transmission capacity is essential. Conventional video encoding technologies, such as encoding of 8K video, have either encoded each frame uniformly regardless of the video content, or divided a frame into several regions and assigned the amount of code according to the characteristics of the video signal components, such as the difficulty of encoding each region.

[0011] Meanwhile, it is expected that ultra-high definition, large screen, and wide viewing angle images exceeding 8K will be used in immersive media. When watching such ultra-high definition, large screen, and wide viewing angle images, the viewer does not watch the entire screen uniformly, but mainly focuses on a part of the image depending on the image content. For this reason, in order to effectively utilize limited transmission capacity, it is not always necessary to uniformly encode the entire screen, and in many cases, the quality of the image content can be maintained high by maintaining high quality only in the region of interest (ROI) of the image where the viewer is focusing. Therefore, when encoding a video signal, it is effective to control the image quality by increasing or decreasing the bit rate according to the region of interest, which is the region of interest that the viewer is focusing on, and to efficiently encode a huge amount of video data.

[0012] However, according to the conventional technology, it is necessary to determine the area where the viewer is watching (gaze area) and notify the encoding device of the information to control the video encoding device. Also, since the gaze area of ​​each viewer is often different, it is necessary to operate the video encoding device using different controls for each viewer's gaze area.

[0013] On the other hand, if the gaze area can be automatically guided in accordance with the video, there will be no need to notify the video encoding device of information indicating the gaze area as additional information; moreover, it can be actively utilized as an element that effectively enhances the viewing experience of video and audio content.

[0014] For example, if a viewer is called or hears a loud sound from a certain direction while watching content, it is a natural reaction of the viewer to turn his / her gaze in the direction of the sound and move the attention area. By using this, a video object in an input video is linked to an audio object, the periphery of the video object in the direction in which the sound from the audio object is generated is determined as the attention area, and encoding control is performed so as to maintain the encoding quality of the attention area higher than other areas, thereby improving the subjective quality.

[0015] In other words, there is a demand for a video encoding device that can guide the area of ​​interest of a video in accordance with video and audio, and encode the video while maintaining high quality of the video content.

[0016] However, in the above-mentioned object-based audio system, a system is being considered in which various information associated with audio objects, such as playback position information, is added as metadata, but this audio object is not linked to a two-dimensional video object and used in video encoding.

[0017] Therefore, in consideration of the above-mentioned problems, an object of the present invention is to provide a video encoding device and program that guides the gaze area of ​​a video in accordance with video and audio, and encodes the video while maintaining high quality of the video content. [Means for solving the problem]

[0018] The video encoding device of the present invention is a video encoding device that encodes video so as to guide a gaze area of ​​the video in accordance with video and audio, and includes a video object position / shape extraction unit that inputs a video before encoding processing and extracts position coordinates and shape coordinates of a video object in a video frame to be processed that constitutes the video from accompanying video metadata including information indicating position coordinates of the video object in the video frame or by video analysis, and generates position coordinates in the video frame to be processed for an audio object that is a sound generator for each video frame to be processed based on audio metadata accompanying the video, and links the audio object to a video object that is located closest based on the position coordinates of the video object, and inputs audio of the audio object in the video frame to be processed or references the video metadata to detect the loudness of the audio object, and generates position coordinates and shape coordinates of the audio object in the video frame to be processed based on the audio metadata. the position coordinates of a video object linked to an audio object in a video frame and information on the volume of the video object; an induced gaze area generation unit which determines a video object to be gaze-guided among the video objects linked as sound sources for each video frame to be processed based on a predetermined criterion using the position coordinates of the video object linked to the audio object, the volume information of the sound, and the shape coordinates of the video object; an induced gaze area generation unit which determines an induced gaze area to surround the shape of the video object; an encoding control information generation unit which controls the amount of code when encoding the video frame to be processed based on the information of the induced gaze area so as to improve the encoding image quality of the induced gaze area more than other areas; and a video encoding unit which encodes the video based on the control of the amount of code, and generates an encoded stream including encoding parameters indicating at least the control information of the amount of code.

[0019] In addition, in the video encoding device of the present invention, the viewing environment on the receiving side of the encoded stream is assumed and determined in advance, and the audio metadata includes position information of a sound source predetermined for each audio object in the video frame to be processed in a virtual space predetermined for the assumed viewing position on the receiving side and the assumed position of the display device.

[0020] In the video encoding device of the present invention, the video objects in the video metadata and the audio objects in the audio metadata are linked in advance before the encoding process.

[0021] In the video encoding device of the present invention, the audio metadata includes information indicating the loudness of an audio object.

[0022] In addition, the video encoding device of the present invention is further equipped with a means that is connected online to a receiving system that receives the encoded stream so as to be able to communicate in two directions, and that acquires information from the receiving system indicating either or both of the actual viewing position and the viewing direction from the actual viewing position, and the induced gaze area generation unit further has a means for determining the induced gaze area by correcting either or both of the assumed viewing position and the direction from the assumed viewing position obtained from the audio metadata.

[0023] Furthermore, the program of the present invention is configured as a program for causing a computer to function as the video encoding device of the present invention. Effect of the Invention

[0024] According to the present invention, not only can the amount of encoded data be compressed without degrading subjective quality, but it also becomes possible to produce content that more strongly reflects the director's intentions and leads to an effective viewing experience. [Brief description of the drawings]

[0025] [Figure 1] 1 is a block diagram showing a schematic configuration of a video encoding device according to an embodiment of the present invention; [Diagram 2] 4 is a flowchart showing a video encoding process in a video encoding device according to an embodiment of the present invention. [Diagram 3] 1A and 1B are diagrams showing examples of information obtained from video metadata and audio metadata, respectively, in a video encoding device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram for explaining the operation of a video encoding device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] (Device configuration) Hereinafter, a configuration of a video encoding device 1 according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing a schematic configuration of the video encoding device 1 according to an embodiment of the present invention. The video encoding device 1 shown in Fig. 1 is a device that links a video object in an input video with an audio object, determines the periphery of the video object in the direction in which the sound of the audio object is generated as a gaze area, and controls encoding so as to maintain the encoding quality of the gaze area higher than other areas, and includes a video object position / shape extraction unit 2, an audio position generation unit 3, an induced gaze area generation unit 4, an encoding control information generation unit 5, and a video encoding unit 6.

[0027] Video object position / shape extraction unit 2 inputs pre-encoding video (a pre-encoding video file shown in units of one video frame or GOP (group of pictures)), extracts the position coordinates and shape coordinates of the video object in the video frame to be processed from accompanying video metadata including information indicating the position coordinates of the video object in the video frame, or by video analysis, and outputs these to audio position generation unit 3 and induced gaze area generation unit 4, respectively. The position coordinates of the video object in the video frame to be processed can be any position that can identify a representative position of the video object, and may be the center of gravity of the video object, for example.

[0028] Here, when accompanying video metadata is input, the video object position / shape extraction unit 2 obtains the position coordinates of the video object in the video frame to be processed from the video metadata, and extracts the shape (shape coordinates) of the video object located at the position coordinates of the video object. When the video object position / shape extraction unit 2 cannot obtain the position coordinates of the video object in the video frame to be processed from the video metadata, it extracts both the position and shape of the video object from the video frame to be processed. Any method can be used to extract the position and shape coordinates of the video object in the video frame to be processed, and the techniques described in Patent Documents 1 and 2, for example, can be used.

[0029] The audio position generating unit 3 inputs audio metadata associated with the video, generates position coordinates in the video frame to be processed for the audio object that is a sound generator based on the audio metadata for each video frame to be processed, selects the video object that is closest to the position coordinates of the audio object by referring to the position coordinates of the video object obtained from the video object position / shape extracting unit 2, and links the audio object to the video object. The audio position generating unit 3 also inputs the audio of the audio object that is a sound generator in the video frame to be processed, detects the loudness of the audio object (for example, average or maximum audio power), and outputs the position coordinates of the video object linked to the audio object in the video frame to be processed and the information on the loudness of the sound to the induced gaze area generating unit 4 in association with each other. However, when the audio metadata includes information indicating the loudness of the audio object, audio input can be omitted.

[0030] In this embodiment, it is assumed that the video object in the video metadata and the audio object in the audio metadata are not linked at the production stage. The audio metadata includes position information of a sound source that is predetermined for each audio object in the video frame to be processed in a virtual space that is predetermined for the expected viewing position of the receiving side and the expected position of the display device. Therefore, the audio position generation unit 3 can generate position coordinates based on the expected viewing position in the video frame to be processed for the audio object that is a sound source based on the audio metadata, and can link the audio object to the video object that is closest based on the position coordinates of the video object. On the other hand, as in the above-mentioned object-based audio system, when the video object in the video metadata and the audio object in the audio metadata are linked in advance at the production stage such as video production by CG (Computer Graphics), the audio position generation unit 3 omits this linking process, and outputs the position coordinates of the video object that is linked in advance to the audio object in the video frame to be processed and the information on the volume of the sound to the induced gaze area generation unit 4 in association with each other.

[0031] The induced gaze area generating unit 4 determines a video object to be gaze-guided among the video objects linked as sound sources based on predetermined criteria such as the content of the sound such as "calling" and the position and size of the video object for each video frame to be processed, based on the position coordinates of the video object linked to the audio object in the video frame to be processed obtained from the audio position generating unit 3 and the information on the volume of the sound, and the shape coordinates of the video object obtained from the video object position and shape extracting unit 2, and determines an induced gaze area so as to surround the shape of the video object, and outputs information on the induced gaze area to the coding control information generating unit 5. Here, "surrounding the shape of the video object" may be a region of the shape of the video object to be gaze-guided, a region tracing several pixels around the shape of the video object, or a predetermined round or angular region including the entire shape of the video object.

[0032] The encoding control information generation unit 5 generates encoding control information based on the information of the induced gaze area obtained from the induced gaze area generation unit 4 so as to improve the encoding image quality of the induced gaze area more than other areas, and controls the amount of code when encoding the video frame to be processed in the video encoding unit 6.

[0033] The video encoding unit 6 encodes the input video on a video frame basis or a GOP basis based on the control of the code amount by the encoding control information generation unit 5, generates an encoded stream including encoding parameters (including encoding control information) that at least indicate the control of the code amount, and transmits the encoded stream to a video decoding device (not shown).

[0034] (device operation) Below, based on Fig. 2, the video encoding process and operation in the video encoding device 1 of this embodiment will be described with reference to Fig. 3 and Fig. 4. Fig. 2 is a flowchart showing the video encoding process in the video encoding device 1 of one embodiment according to the present invention. Also, Figs. 3(a) and (b) are diagrams showing examples of information obtained from video metadata and audio metadata, respectively, in the video encoding device 1 of one embodiment according to the present invention. And Fig. 4 is a diagram explaining the operation in the video encoding device 1 of one embodiment according to the present invention.

[0035] 2, first, in the video encoding device 1 of this embodiment, a video object position / shape extraction unit 2 inputs a pre-encoding video represented in units of one video frame or GOP (group of pictures) (step S1), and extracts the position coordinates and shape coordinates of a video object in a video frame to be processed from accompanying video metadata including information indicating the position coordinates of the video object in the video frame or by video analysis (step S2). The position coordinates of a video object in a video frame to be processed may be any position that identifies a representative position of the video object, and may be, for example, the center of gravity of the video object.

[0036] Here, when the video object in the video metadata and the audio object in the audio metadata are linked in advance before the encoding process, it is not necessary to assume and determine the viewing environment on the receiving side in advance. In this embodiment, however, as shown in Fig. 4, the viewing environment on the receiving side of the encoded stream generated by the video encoding device 1 is assumed and determined in advance. For example, in world coordinates (X, Y, Z) with the assumed viewing position as the origin, the position of the display device (display) from the assumed viewing position and the size (H, V) of the video frame Fn corresponding to the display device (display) are defined. That is, the position coordinates of the video frame Fn are also defined on the world coordinates (X, Y, Z). Note that the example shown in Fig. 4 is just an example, and the definitions of the assumed viewing position and the world coordinates (X, Y, Z) can be determined arbitrarily.

[0037] For this reason, when the video object position / shape extraction unit 2 receives the associated video metadata, it obtains the position coordinates of the video object in the video frame to be processed from the video metadata, and extracts the shape (shape coordinates) of the video object located at the position coordinates of the video object. As shown in Fig. 3(a), the video metadata can be such that, for each predefined video object in a certain video frame Fn (n indicates the frame number), the position coordinates of each video object can be extracted within the range of the size (H, V) of the video frame Fn, such that the barycentric coordinates of video object number #1 are (h1, v1), and the barycentric coordinates of video object number #2 are (h2, v2).

[0038] Furthermore, when the position coordinates of a video object in a video frame to be processed cannot be obtained from the video metadata, the video object position / shape extraction unit 2 extracts both the position and shape of the video object from the video frame to be processed. In this case, a video object number that identifies each video object in the video frame Fn is provisionally set.

[0039] Next, the video encoding device 1 of this embodiment inputs audio metadata associated with the video by the audio position generation unit 3, generates position coordinates in the video frame to be processed for the audio object that is a sound generator based on the audio metadata for each video frame to be processed, and links it to the video object at the nearest position based on the position coordinates of the video object obtained from the video object position / shape extraction unit 2, inputs the audio of the audio object that is a sound generator in the video frame to be processed, detects the loudness of the audio object (for example, average or maximum audio power), and associates the position coordinates of the video object linked to the audio object in the video frame to be processed with the information on the loudness of the audio (step S3). However, when the audio metadata includes information indicating the loudness of the audio object, audio input can be omitted.

[0040] In this embodiment, an example will be described assuming that the video object in the video metadata and the audio object in the audio metadata are not linked at the production stage. The audio metadata in this case includes position information of the sound source predefined for each audio object in the video frame to be processed in a virtual space predefined for the expected viewing position of the receiving side and the expected position of the display device. Therefore, as shown in FIG. 3(b), the sound source direction can be specified based on the predefined expected viewing position for each audio object in the video frame to be processed. The audio metadata defines the position of the display from the expected viewing position in world coordinates (X, Y, Z) for each predefined audio object in a certain video frame Fn, for example, so that the sound source direction based on the expected viewing position of audio object number #1 is (x1, y1, z1) and the sound source direction based on the expected viewing position of audio object number #2 is (x2, y2, z2), and therefore the position coordinates of each audio object in a certain video frame Fn can be extracted.

[0041] Then, the audio position generating unit 3 can select a video object having the position coordinates of the closest position to the position coordinates of each audio object in a certain video frame Fn, and link the audio object as the sound generator to the video object. Here, as shown in FIG. 4, for convenience of explanation, video object #1 is linked to audio object #1, and video object #1 is linked to audio object #1. That is, the audio position generating unit 3 selects a video object having the position coordinates of the closest position to the position coordinates of each audio object in a certain video frame Fn, whether it is a provisionally set video object number or a predefined video object number, and can link the audio object as the sound generator to the video object. Note that the number of objects to be linked is not limited to one, and may be multiple.

[0042] The audio position generation unit 3 detects the loudness (e.g., average or maximum sound power) of the audio object from the input audio, and can therefore associate the position coordinates of the video object linked to the audio object in the video frame to be processed with the loudness information. On the other hand, as in the above-mentioned object-based audio system, when the video object in the video metadata and the audio object in the audio metadata are linked in advance at the production stage, the audio position generation unit 3 can omit this linking process and associate the position coordinates of the video object linked in advance to the audio object in the video frame to be processed with the loudness information.

[0043] Next, in the video encoding device 1 of this embodiment, based on the position coordinates and sound volume information of the video object linked to the audio object in the video frame to be processed obtained from the audio position generation unit 3, and the shape coordinates of the video object obtained from the video object position / shape extraction unit 2, the induced gaze area generation unit 4 determines, based on predetermined criteria, for each video frame to be processed, a video object to be gaze-induced from among the video objects linked as sound sources, and determines an induced gaze area to surround its shape (step S4).

[0044] For example, the guiding gaze area generating unit 4 ranks which video objects are likely to be gazed at based on predetermined criteria such as the contents of the voice such as "calling out" and the position and size of the video object, and determines the video objects within a predetermined rank (for example, within two as shown in FIG. 4) from the highest ranked ones determined to be likely to be gazed at as the guiding gaze area for guiding the gaze (determining the area to surround the shape of the video object). Note that in FIG. 4, the guiding gaze area is a predetermined angular area including the entire shape of the video object, but it may be an area of ​​the shape of the video object itself to be gazed at, an area tracing several pixels around the shape of the video object, or a predetermined circular area including the entire shape of the video object. In this way, the guiding gaze area generating unit 4 can determine which video object to guide the gaze to. In addition, it is preferable to limit the number of guiding attention areas to a predetermined number, such as one or more, in order to reduce the amount of coding, and when multiple guiding attention areas have overlapping areas, it is preferable to re-define them as a single guiding attention area in terms of processing load and quality.

[0045] Next, the video encoding device 1 of this embodiment controls the code amount when encoding the video frame to be processed in the video encoding unit 6, based on the information of the induced gaze area obtained from the induced gaze area generation unit 4, so as to improve the encoded image quality of the induced gaze area more than other areas, using the encoding control information generation unit 5. Then, based on the control of the code amount by the encoding control information generation unit 5, the video encoding unit 6 encodes the input video in units of one video frame or GOP, generates an encoded stream including encoding parameters (including encoding control information) indicating at least control information of the code amount, and transmits it to a video decoding device (not shown) (step S5).

[0046] In this way, the coding control information generator 5 controls the amount of code so as to improve the coding quality of the guided attention region more than other regions. Specifically, there is a method of allocating a larger amount of code to the attention region.

[0047] The video encoding unit 6 then encodes the input video using the code amount controlled by the encoding control information generation unit 5, and outputs the encoded stream together with the encoding parameters to the outside. Any encoding method may be used, but the code amount (quantization parameter value) must be able to be changed depending on the position within the screen. The quantization parameter value can be transmitted as the encoding parameter. The area to be changed must be in units of blocks or the like according to the encoding process.

[0048] Although not shown in the figure, the video decoding device that receives and decodes the encoded stream transmitted from the video encoding device 1 of this embodiment may be configured to decode the encoded stream based on the encoding parameters obtained from the video encoding device 1, and a device similar to that used for existing video decoding processing may be used.

[0049] In this way, the video encoding device 1 of this embodiment is configured to determine and encode an induced gaze area for video encoding based on audio information (audio metadata and audio input), thereby reducing the amount of encoded data while maintaining subjective quality, and providing an effective viewing experience by more strongly reflecting the production intent on the receiving side.

[0050] Therefore, the video encoding device 1 of the present invention not only makes it possible to compress the amount of encoded data without degrading subjective quality, but also makes it possible to produce content that more strongly reflects the director's intentions and leads to an effective viewing experience.

[0051] The video encoding device 1 according to the present invention can be configured by a computer, and a program for making each processing unit of the video encoding device 1 function can be suitably used. Specifically, a control unit for controlling each processing unit of the video encoding device 1 can be configured by a central processing unit (CPU) in a computer, and a storage unit for appropriately storing a program required for operating each processing unit can be configured by at least one memory. That is, the function of each processing unit of the video encoding device 1 can be realized by having the CPU execute the program on such a computer. Furthermore, a program for making each processing unit of the video encoding device 1 function can be stored in a predetermined area of ​​the storage unit (memory) described above. Such a storage unit can be configured by a RAM or ROM inside the device, or can be configured by an external storage device (for example, a hard disk). Moreover, such a program can be configured as a part of software (stored in a ROM or an external storage device) on an OS used by the computer. Furthermore, a program for making such a computer function as each processing unit of the video encoding device 1 can be recorded on a computer-readable recording medium. Moreover, each processing unit of the video encoding device 1 can be configured as a part of hardware or software, and each can be realized by combining them.

[0052] Although the present invention has been described above by taking specific examples of embodiments, the present invention is not limited to the above-mentioned examples of embodiments, and various modifications are possible without departing from the technical concept of the present invention. For example, in the above-mentioned embodiment, an example in which the sound level of each sound object is detected based on sound input has been mainly described, but by making the sound metadata describe the sound level of each sound object, it is possible to detect a video object that is a sound-producing body without taking into account sound input.

[0053] Furthermore, the video encoding device 1 according to the present invention can be configured to be connected online to enable two-way communication with a receiving system (including a video decoder, a display device, and peripheral devices) that receives the encoded stream generated by the video encoding device 1. In this case, the video encoding device 1 further includes a means for acquiring information indicating either or both of the "actual viewing position" and the "viewing direction from the actual viewing position" by using an upstream line from the receiving system to the video encoding device 1. The induced gaze area generating unit 4 in the video encoding device 1 can also determine the induced gaze area by correcting either or both of the "assumed viewing position" and the "direction from the assumed viewing position" obtained from the audio metadata.

[0054] For example, in a configuration in which the video encoding device 1 according to the present invention and a receiving system (including a video decoding device, a display device, and peripheral devices) are connected online to enable two-way communication, a configuration in which VR (Virtual Reality) goggles are used as peripheral devices in the receiving system may be adopted. In this case, when the gaze direction is obtained from an inertial measurement unit (IMU) such as an acceleration sensor and a gyroscope provided in the VR goggles, the induced gaze area generating unit 4 in the video encoding device 1 can determine the induced gaze area by correcting the "direction from the assumed viewing position" obtained from the audio metadata.

[0055] Furthermore, in a configuration in which the video encoding device 1 according to the present invention and a receiving system (including a video decoding device, a display device, and peripheral devices) are connected online to enable two-way communication, a receiving environment sensor that measures the relative positional relationship between the display device and the actual viewing position can be provided as a peripheral device in the receiving system. In this case, the induced gaze area generating unit 4 in the video encoding device 1 can be configured to measure the actual viewing position with this receiving environment sensor and transmit information indicating the actual viewing position to the video encoding device 1 side using an uplink, thereby correcting the "assumed viewing position" obtained from the audio metadata to determine the induced gaze area. [Industrial Applicability]

[0056] According to the present invention, not only can the amount of encoded data be compressed without degrading subjective quality, but it also makes it possible to produce content that more strongly reflects the director's intentions and leads to an effective viewing experience, making it useful for encoding and transmitting video that has the characteristics of ultra-high definition, large screen, and wide viewing angle. [Explanation of symbols]

[0057] 1. Video Encoding Device 2. Video object position and shape extraction unit 3. Audio position generator 4 Guided gaze area generation unit 5. Encoding control information generator 6 Video Encoding Section

Claims

1. A video encoding device that encodes a video so as to guide a gaze area of ​​the video according to the video and audio, a video object position / shape extraction unit that receives a video before encoding processing and extracts position coordinates and shape coordinates of a video object in a video frame to be processed that constitutes the video from associated video metadata including information indicating position coordinates of the video object in the video frame or by video analysis; an audio position generation unit which generates position coordinates in the video frame to be processed for an audio object that is a sound source for each video frame to be processed based on audio metadata accompanying the video, links the audio object to a nearest video object based on the position coordinates of the video object, inputs the audio of the audio object in the video frame to be processed or detects the loudness of the audio object by referring to the video metadata, and associates the position coordinates of the video object linked to the audio object in the video frame to be processed with information on the loudness of the audio object; an induced gaze area generating unit which, based on the position coordinates of the video object associated with the audio object, information on the volume of the sound, and the shape coordinates of the video object, determines a video object to be gaze-induced from among the video objects associated as sound sources for each video frame to be processed, based on a predetermined criterion, and determines an induced gaze area to surround the shape of the video object; a coding control information generating unit that controls a code amount when coding a video frame to be processed based on information of the induced attention area so as to improve the coding image quality of the induced attention area more than other areas; a video encoding unit that encodes the video based on the control of the code amount and generates an encoded stream including encoding parameters that indicate at least control information of the code amount; A video encoding device comprising:

2. A viewing environment on a receiving side of the encoded stream is assumed and determined in advance, The video encoding device described in claim 1, characterized in that the audio metadata includes position information of a sound source predetermined for each audio object in the video frame to be processed in a virtual space predetermined for the expected viewing position of the receiving side and the expected position of the display device.

3. 2. The video encoding device according to claim 1, wherein the video object in the video metadata and the audio object in the audio metadata are linked in advance before the encoding process.

4. 4. The video encoding device according to claim 1, wherein the audio metadata includes information indicating a volume of an audio object.

5. The apparatus further includes a means for acquiring information indicating either or both of an actual viewing position and a viewing direction from the actual viewing position, the information being connected online to a receiving system that receives the encoded stream so as to be capable of bidirectional communication, from the receiving system side; The video encoding device according to claim 1 , wherein the induced gaze area generation unit further comprises means for determining the induced gaze area by correcting either or both of the assumed viewing position and the direction from the assumed viewing position obtained from the audio metadata.

6. A program for causing a computer to function as the video encoding device according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Object detection and extraction device for image

    JP2001307091A

  • Apparatus, method, and program for video related content generation

    JP2004015523A

  • Image processor and image processing method

    JP2010219786A

  • Video sound processor, and video sound processing method, and program

    JP2011071686A

  • Image processing system and image processing method

    JP2013168739A

Cited By

  • Video-derived audio processing

    US12231865B2