Video capture system

Abstract

The application discloses a video acquisition system, and relates to the technical field of head-mounted devices. The video acquisition system comprises a head-mounted device, a processing device, and a third-party live broadcast platform. The head-mounted device is used for acquiring image data and audio data of a live broadcast object, obtaining a video stream and an audio stream respectively, encoding the video stream and the audio stream respectively, obtaining a video encoded stream and an audio encoded stream, and transmitting the video encoded stream and the audio encoded stream to the processing device. The processing device is used for receiving the video encoded stream and the audio encoded stream transmitted by the head-mounted device, decoding the video encoded stream and the audio encoded stream respectively, obtaining the video stream and the audio stream, performing an image enhancement operation on the video stream to obtain an enhanced video stream, and pushing the enhanced video stream and the audio stream to the third-party live broadcast platform. Based on the video acquisition system, the head-mounted device can realize a live broadcast function, which expands the function of the head-mounted device.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and more specifically, to a video capture system. Background Technology

[0002] With rapid economic and technological development, the application scenarios of head-mounted devices continue to expand, deeply penetrating multiple fields such as entertainment, education, healthcare, and industry. Further expanding their functionality will not only help improve the practicality and intelligence of the devices but also optimize cost-effectiveness by enhancing the user experience.

[0003] Therefore, how to further expand the functionality of head-mounted devices has become an urgent technical problem to be solved. Summary of the Invention

[0004] One objective of this application is to provide a new technical solution for video capture.

[0005] According to a first aspect of this application, a video capture system is provided, comprising:

[0006] A head-mounted device is used to collect image data and audio data of a live broadcast subject, obtain video streams and audio streams respectively, encode the video streams and audio streams respectively, obtain video encoded streams and audio encoded streams, and transmit the video encoded streams and audio encoded streams to a processing device;

[0007] The processing device is configured to receive the video encoded stream and the audio encoded stream sent by the head-mounted device, decode the video encoded stream and the audio encoded stream respectively to obtain the video stream and the audio stream, perform image enhancement operation on the video stream to obtain the enhanced video stream, and push the enhanced video stream and the audio stream to a third-party live streaming platform.

[0008] Optionally, encoding the video stream and the audio stream respectively to obtain a video encoded stream and an audio encoded stream includes:

[0009] Based on the first image and the second image in the video stream, the video scene type is determined. The video scene type includes static scenes and dynamic scenes. The first image is the frame preceding the second image.

[0010] Based on the video scene type, the parameter values ​​of the encoding parameters of the video encoder are set to parameter values ​​that match the video scene type;

[0011] The video encoder is controlled to encode the first image and the image of a preset number of subsequent frames to obtain a partial video encoded stream;

[0012] The step of determining the video scene type based on the first and second images in the video stream is repeated, taking the next frame of a preset number of frames starting from the first image as the first image.

[0013] Optionally, determining the video scene type based on the first and second images in the video stream includes:

[0014] Based on the first and second images in the video stream, determine the corner motion quantization value and / or motion region quantization value;

[0015] The video scene type is determined based on the corner motion quantization value and / or motion region quantization value;

[0016] Wherein, the corner motion quantization value is the quantization value of the motion vector formed by the corner point in the second image moving from the corresponding position in the first image to the corresponding position in the second image, and the motion region quantization value is the quantization value of the region size of the pixel-formed area that describes the motion of the second image compared to the first image.

[0017] Optionally, before determining the video scene type based on the first and second images in the video stream, the head-mounted device is further configured to:

[0018] The first image and the second image in the video stream are respectively subjected to grayscale processing to obtain a first grayscale image corresponding to the first image and a second grayscale image corresponding to the second image;

[0019] Update the first image to the first grayscale image;

[0020] Update the second image to the second grayscale image.

[0021] Optionally, determining the corner motion quantization value and / or motion region quantization value based on the first image and the second image in the video stream includes:

[0022] Based on the second image in the video stream, determine the first position of each corner point in the second image;

[0023] For any of the corner points, determine the second position of the corner point in the first image from the first image;

[0024] The motion quantization value of the corner point is determined based on the first position of the corner point in the second image and the second position of the corner point in the first image.

[0025] Optionally, determining the corner motion quantization value and / or motion region quantization value based on the first image and the second image in the video stream includes:

[0026] A difference image is determined based on the first image and the second image in the video stream, wherein the pixel value of any pixel in the difference image is the absolute value of the difference between the pixel values ​​of the corresponding pixels in the first image and the second image;

[0027] The difference image is binarized according to a preset threshold to obtain a binarized image. The preset threshold is the minimum change in pixel value caused by the movement of an object in the image.

[0028] Based on the binarized image, determine the quantization value of the motion region.

[0029] Optionally, determining the video scene type based on the corner motion quantization value and / or motion region quantization value includes:

[0030] If the corner motion quantization value is greater than or equal to the preset corner motion change amount, the video scene type is determined to be a dynamic scene;

[0031] Alternatively, if the motion region quantization value is greater than or equal to the preset motion region quantization value, the video scene type is determined to be a dynamic scene;

[0032] Alternatively, normalization processing can be performed on the corner motion quantization value and the motion region quantization value respectively to obtain normalized corner motion quantization value and normalized motion region quantization value;

[0033] The motion quantization value is determined based on the normalized corner motion quantization value and the normalized motion region quantization value;

[0034] If the motion quantization value is greater than or equal to the preset motion quantization value, the video scene type is determined to be a dynamic scene.

[0035] Optionally, the encoding parameters include at least one of frame rate, bit rate, and GOP, and setting the parameter values ​​of the video encoder's encoding parameters to match the video scene type according to the video scene type includes:

[0036] When the video scene type is a static scene, the frame rate of the video encoder is set to a first frame rate, the bit rate to a first bit rate, and the GOP to a first GOP;

[0037] When the video scene type is a dynamic scene, the frame rate of the video encoder is set to a second frame rate, the bit rate is set to a second bit rate, and the GOP is set to a second GOP at least one of the following:

[0038] Wherein, the first frame rate is less than the second frame rate, the first bitrate is less than the second bitrate, and the first GOP is greater than the second GOP.

[0039] Optionally, the processing device is further configured to receive a live broadcast control command and send the live broadcast control command to the head-mounted device;

[0040] The head-mounted device is also configured to receive the live streaming control command, and in response to the live streaming control command, execute live streaming control that matches the live streaming control command, the live streaming control command including any one of: start live streaming, stop live streaming, pause live streaming, and resume live streaming.

[0041] Optionally, transmitting the video encoded stream and audio encoded stream to the processing device includes:

[0042] The video and audio encoded streams are transmitted to the processing device according to a preset traffic shaping strategy and / or a preset congestion control strategy.

[0043] Optionally, the head-mounted device is also configured to send data streams other than the video encoded stream and the audio encoded stream to the processing device;

[0044] The step of transmitting the video encoded stream and the audio encoded stream to the processing device includes: when at least two of the data stream, the video encoded stream and the audio encoded stream need to be sent to the processing device at the same time, sending at least two of the data stream, the video encoded stream and the audio encoded stream according to a set priority;

[0045] The priority is set such that the priority of the data stream is higher than the priority of the audio encoded stream, and the priority of the audio encoded stream is higher than the priority of the video encoded stream.

[0046] Optionally, the head-mounted device is further configured to store the video stream in a first memory and the audio stream in a second memory;

[0047] The process of encoding the video stream and the audio stream respectively to obtain a video encoded stream and an audio encoded stream includes:

[0048] The video encoder is controlled to read the video stream from the first memory and perform video encoding operations on the video stream to obtain a video encoded stream;

[0049] The audio encoder is controlled to read the audio stream from the second memory, and an audio encoding operation is performed on the audio stream to obtain an audio encoded stream;

[0050] The head-mounted device is also configured to read the video stream from the first memory and the audio stream from the second memory, and perform a preview operation on the read video stream and audio stream.

[0051] This application provides a video acquisition system, including: a head-mounted device for acquiring image data and audio data of a live-streaming subject, obtaining video and audio streams respectively, encoding the video and audio streams respectively, obtaining encoded video and encoded audio streams, and transmitting the encoded video and encoded audio streams to a processing device; and a processing device for receiving the encoded video and encoded audio streams sent by the head-mounted device, decoding the encoded video and encoded audio streams respectively, obtaining video and audio streams, performing image enhancement operations on the video streams to obtain enhanced video streams, and pushing the enhanced video and audio streams to a third-party live-streaming platform. Based on the video acquisition system provided by this application, the head-mounted device can realize live-streaming functionality, which expands the functionality of the head-mounted device.

[0052] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0054] Figure 1 This is a schematic diagram of the structure of a video acquisition system provided in this application;

[0055] Figure 2 This is a flowchart of the execution steps of a head-mounted device in a video acquisition system provided in this application;

[0056] Figure 3 This is a flowchart of the processing device execution steps in a video acquisition system provided in this application. Detailed Implementation

[0057] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this application.

[0058] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0059] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0060] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0061] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0062] This application provides a video capture system 100, such as... Figure 1 As shown, the device includes a head-mounted device 120 and a processing device 110. The head-mounted device 120 can be an AR device, a MR device, or a VR device, etc. The head-mounted device 120 can be an all-in-one head-mounted device or a split-type head-mounted device. The processing device 110 can be a terminal or a server. The terminal can be a smartphone, laptop, tablet, PDA, etc. If the head-mounted device 120 is a split-type head-mounted device, the terminal can also be a control device for the split-type head-mounted device, such as a Puck device. The server can be a cloud server, etc.

[0063] For the head-mounted device 120 in the video capture system, such as Figure 2 As shown, it is used to perform the following steps S210 to S230.

[0064] Step S210: Collect image data and audio data of the live streaming object to obtain video stream and audio stream respectively.

[0065] In this embodiment, a user wears a head-mounted device 120 to view the live stream from a first-person perspective. The image acquisition device in the head-mounted device 120 acquires image data of the live stream object, and further generates a video stream based on the acquired image data. Additionally, the microphone in the head-mounted device 120 acquires audio data of the live stream object, and further generates an audio stream based on the acquired audio data.

[0066] Since the user wearing the head-mounted device 120 views the live stream from a first-person perspective, the head-mounted device 120 captures image data of the live stream from this perspective, allowing viewers to experience the live stream from a first-person viewpoint. Furthermore, compared to traditional live streaming devices with fixed perspectives, the user of the head-mounted device 120 in the video capture system provided in this application can adjust the perspective of the head-mounted device 120 based on the behavior of the live stream subject. For example, when introducing live stream subject B, the viewpoint can be moved from live stream subject A to live stream subject B. Therefore, live streaming using the head-mounted device 120 in the video capture system provided in this application improves the viewing experience for viewers of the live stream subject.

[0067] Step S220: Encode the video stream and the audio stream respectively to obtain the video encoded stream and the audio encoded stream.

[0068] In one embodiment of this application, an encoder using the H.265 encoding format can be used to encode the video stream to obtain a video encoded stream. Similarly, an encoder using the AAC encoding format can be used to encode the audio stream to obtain an audio encoded stream. Of course, encoders using other encoding formats can also be used to encode the video and audio streams; this application does not limit the scope of the application.

[0069] Step S230: The video encoded stream and the audio encoded stream are transmitted to the processing device 110.

[0070] In this embodiment, the head-mounted device 120 is communicatively connected to the processing device 110, and the head-mounted device 120 transmits video and audio encoded streams to the processing device 110 based on this communication connection. In one example, the communication connection can be one of the following: USB connection, Bluetooth connection, or WIFI connection.

[0071] In one embodiment of this application, in order to improve the stability and reliability of the transmission of video and audio encoded streams from the head-mounted device 120 to the processing device 110, the above-mentioned step S230 can be specifically implemented by the following step S2301. That is, the head-mounted device 120 is specifically used to perform the following step S2301.

[0072] Step S2301: According to the preset traffic shaping strategy and / or preset congestion control strategy, the video encoded stream and audio encoded stream are transmitted to the processing device 110.

[0073] In this embodiment, the preset traffic shaping strategy is a preset strategy that controls the data packet sending rate to ensure that the traffic conforms to preset bandwidth limits or service level agreements (SLAs), thereby avoiding network congestion caused by sudden traffic surges. The preset congestion control strategy is a preset strategy that dynamically adjusts the sending rate to prevent network performance degradation due to excessive data injection, ensuring that the communication connection can still operate stably under high load. This application does not limit the specific content of the preset traffic shaping strategy and the preset congestion control strategy.

[0074] Through the above step S2301, the head-mounted device can stably and reliably transmit the video and audio encoded streams to the processing device 110.

[0075] In one embodiment of this application, the head-mounted device 120 is also used to perform the following step S240.

[0076] Step S240: Send a data stream other than the video encoded stream and the audio encoded stream to the processing device 110.

[0077] In one example, head-mounted device 120 sends a data stream other than the video and audio encoded streams to processing device 110 as notification data to inform processing device 110 that the current battery is low.

[0078] Based on step S240 above, there may be a situation where at least two of the data stream, video encoded stream, and audio encoded stream need to be sent to the processing device 110 at the same time. To avoid data loss in this case and to make full use of the bandwidth of the communication connection, step S230 above is specifically implemented through step S2302 below.

[0079] In step S2302, if at least two of the data stream, video encoded stream, and audio encoded stream are sent to the processing device 110 at the same time, at least two of the data stream, video encoded stream, and audio encoded stream are sent according to the set priority.

[0080] The priority is set so that the data stream has a higher priority than the audio encoded stream, and the audio encoded stream has a higher priority than the video encoded stream.

[0081] In one embodiment of this application, the head-mounted device 120 transmits data streams, video encoded streams, and audio encoded streams to the processing device 110 in the form of data packets. A corresponding priority identifier can be set in the priority field of the data packets; for example, "0" in the priority field indicates the lowest priority, and "7" indicates the highest priority.

[0082] In this embodiment, a priority that is inversely proportional to the size of the data to be transmitted is set, which can ensure the quality of data transmission as much as possible.

[0083] In one embodiment of this application, the head-mounted device 120 is also used to perform the following step S250.

[0084] Step S250: Store the video stream in the first memory and the audio stream in the second memory.

[0085] Based on step S250 above, step S220 above can be specifically implemented through the following steps S2201 and S2202.

[0086] Step S2201: Control the video encoder to read the video stream from the first memory, perform video encoding operation on the video stream, and obtain the video encoded stream.

[0087] Step S2202: Control the audio encoder to read the audio stream from the second memory, perform audio encoding operation on the audio stream, and obtain the audio encoded stream.

[0088] Based on the above steps S2201 and S2202, the head-mounted device 120 is also used to perform the following step S260.

[0089] Step S260: Read the video stream from the first memory and the audio stream from the second memory, and perform a preview operation on the read video stream and audio stream.

[0090] In this embodiment, the head-mounted device 120 stores the obtained video stream in a first memory and extracts the video stream from the first memory for encoding, and stores the obtained audio stream in a second memory and extracts the audio stream from the second memory for encoding. Furthermore, when performing a preview operation, the head-mounted device 120 extracts the video stream from the first memory and the audio stream from the second memory. This means that the encoding and preview operations of the head-mounted device 120 share the first and second memory, avoiding the need to allocate separate memory for encoding and preview operations and copy data from the encoding memory to the preview memory. This further improves the data processing speed of the head-mounted device 120.

[0091] In one embodiment of this application, in order to effectively utilize limited bandwidth resources and achieve high-quality video transmission, the encoding of the video stream to obtain the video encoded stream in step S220 is specifically implemented through the following steps S2203 to S2206.

[0092] Step S2203: Determine the video scene type based on the first image and the second image in the video stream. The video scene type includes static scenes and dynamic scenes. The first image is the frame preceding the second image.

[0093] In this embodiment, the first image is the first frame of the unencoded video stream, and the second image is the frame following the first image.

[0094] A static scene refers to a scene where, compared to the first image, the objects in the second image are relatively stationary; that is, the second image is essentially the same as the first image. An example of a static scene is one where the position of the image acquisition device does not change, and there are no significantly moving objects in the environment captured by the image acquisition device.

[0095] Correspondingly, a dynamic scene refers to an image in which objects are in relative motion compared to the first image, meaning the second image changes compared to the first image. Examples of dynamic scenes include: the image acquisition device acquiring the first image and then acquiring the second image after its position changes; or, the image acquisition device's position remains unchanged, but there are moving objects in the environment it captures; or, the image acquisition device's position changes, but the movement of objects in the environment it captures is asynchronous.

[0096] In one embodiment of this application, the specific implementation of step S2203 above can be: generating an input instruction for the AI ​​big data model, such as "Please determine whether the video scene type corresponding to the first image and the second image is a dynamic scene or a static scene?"; inputting the aforementioned input instruction, the first image, and the second image into the AI ​​big data model, and having the AI ​​big data model output the video scene type.

[0097] Step S2204: Based on the video scene type, set the parameter values ​​of the encoding parameters of the video encoder to parameter values ​​that match the video scene type.

[0098] In one embodiment of this application, the encoding parameters of the video encoder are at least one of frame rate, bit rate, and GOP. GOP is used to describe the image frame interval.

[0099] Furthermore, the specific implementation of step S2204 above is as follows: based on the video scene type, determine the parameter values ​​of the encoder's encoding parameters that match the video scene type; set the encoder's parameter values ​​to the previously determined parameter values.

[0100] Taking a video encoder with at least one of frame rate, bit rate, and GOP as an example, when the video scene type is static, the parameter values ​​matching the video scene type are those that can encode a video stream satisfying at least one of low frame rate, low bit rate, and high GOP. This is because in static scenes, the image change rate is relatively low. When encoding video using a video encoder with parameter values ​​that can encode a video stream satisfying at least one of low frame rate, low bit rate, and high GOP, the smoothness and clarity of the encoded video stream will not be significantly affected, i.e., the quality of the encoded video stream will not be significantly affected, and bandwidth resources used by some of the encoded video stream during transmission can be saved. Conversely, in dynamic scenes, the parameter values ​​matching the video scene type are those that can encode a video stream satisfying at least one of high frame rate, high bit rate, and low GOP. In this way, when video encoding is performed by a video encoder that can encode video streams that meet at least one of the following parameters: high frame rate, high bit rate, and low GOP, the smoothness and clarity of the encoded video stream can be guaranteed, that is, the quality of the encoded video stream can be guaranteed.

[0101] Based on the above content, step S2204 is specifically implemented through the following steps S22041 and S22042.

[0102] S22041, when the video scene type is a static scene, set the video encoder frame rate to the first frame rate, bit rate to the first bit rate, and GOP to the first GOP at least one of these.

[0103] Step S22042: When the video scene type is a dynamic scene, set the video encoder's frame rate to at least one of the following: second frame rate, second bitrate, and GOP to the second GOP.

[0104] Among them, the first frame rate is less than the second frame rate, the first bitrate is less than the second bitrate, and the first GOP is greater than the second GOP.

[0105] In this embodiment, the first frame rate, the second frame rate, the first bitrate, the second bitrate, the first GOP, and the second GOP can be set based on experience.

[0106] Step S2205: Control the video encoder to encode the first image and the image with a preset number of subsequent frames of the first image to obtain a partial video encoded stream.

[0107] In this embodiment, the video stream obtained by encoding the first image and the image with a preset number of subsequent frames by the video encoder is referred to as the partial video encoded stream.

[0108] The first image and the subsequent preset number of frames total the preset number of frames plus one frame. This preset number of frames plus one frame corresponds to the number of frames where the image change rate is less than or equal to the preset change rate. When the image change rate is less than the preset change rate, the video scene type corresponding to the first image and the subsequent preset number of frames is considered unchanged.

[0109] The preset frame number plus 1 frame corresponds to the frame rate of the images in the video, and can be set based on experience, for example, 10 frames.

[0110] Step S2206: Take the next frame of the preset number of frames starting from the first image as the first image, and repeat the step of determining the video scene type based on the first image and the second image in the video stream.

[0111] In this embodiment, the complete encoding of the video stream can be achieved through the above step S2206, resulting in a complete video encoded stream.

[0112] Furthermore, in one embodiment of this application, if the first image and its subsequent preset frame number image are images of a first format, and the video encoder is used to encode the image of a second format, the specific implementation of step S2205 is as follows: converting the first image of the first format and its subsequent preset frame number image of the first image into the first image of the second format and its subsequent preset frame number image of the first image; controlling the video encoder to encode the first image of the second format and its subsequent preset frame number image of the first image to obtain a partial video encoded stream. In one example, the first format is RGB format and the second format is YUV format.

[0113] In one embodiment of this application, step S2203 can also be implemented by the following steps S22031 and S22032.

[0114] Step S22031: Determine the corner motion quantization value and / or motion region quantization value based on the first image and the second image in the video stream.

[0115] The corner motion value is the quantized value of the motion vector formed by the movement of a corner point in the second image from its corresponding position in the first image to its corresponding position in the second image. The motion region quantization value is the quantized value of the region size describing the area formed by pixels that produce motion in the second image compared to the first image.

[0116] In one embodiment of this application, in order to reduce the amount of data in the first image and the second image, and to enhance the feature information in the first image and the second image, based on the embodiments shown in steps S22031 and S22032, the video encoding method provided by this application further includes the following steps S22033 and S22035 before the above step S2203.

[0117] Step S22033: Perform grayscale processing on the first image and the second image in the video stream respectively to obtain the first grayscale image corresponding to the first image and the second grayscale image corresponding to the second image.

[0118] Step S22034: Update the first image to the first grayscale image.

[0119] Step S22035: Update the second image to a second grayscale image.

[0120] In this embodiment, when the first image and the second image are YUV format images, the specific implementation of the above step S22034 is as follows: retain the value of the Y channel of each pixel in the first image to form a first grayscale image; similarly, retain the value of the Y channel of each pixel in the second image to form a second grayscale image.

[0121] When both the first and second images are RGB format images, the specific implementation of step S22034 is as follows: For the pixel at position a in the first image, the grayscale value of the pixel at position a is determined according to the calculation formula for converting RGB pixels to grayscale pixels, and used as the pixel value of the pixel at position a in the first grayscale image. Based on this, the first grayscale image corresponding to the first image is obtained. Similarly, the second grayscale image corresponding to the second image can be obtained. Specifically, the calculation formula for converting RGB pixels to grayscale pixels can be gray = (R + G + B) / 3, where R is the specific value of the R channel of the pixel, G is the specific value of the G channel of the pixel, B is the specific value of the B channel of the pixel, and gray is the grayscale value of the pixel.

[0122] After obtaining the first grayscale image, the first image is updated to the first grayscale image. Similarly, after obtaining the second grayscale image, the second image is updated to the second grayscale image. It is understood that in this embodiment, after step S22035 above, the first image in the video encoding method provided by this application is essentially the first grayscale image, and the second image is essentially the second grayscale image.

[0123] In this embodiment, step S22031 includes determining the corner motion quantization value based on the first image and the second image in the video stream, and / or determining the motion region quantization value based on the first image and the second image in the video stream. The step of determining the corner motion quantization value based on the first image and the second image in the video stream is implemented through the following steps S22031-1 to S22031-3.

[0124] Step S22031-1: Based on the second image in the video stream, determine the first position of each corner point in the second image.

[0125] In one embodiment of this application, step S22031-1 is specifically as follows: extracting each corner point in the second image using the Harris corner detection algorithm; for any corner point, determining the position of the corner point in the second image, and recording it as the first position of the corner point in the second image.

[0126] Step S22031-2: For any corner point, determine the second position of the corner point in the first image from the first image.

[0127] In one embodiment of this application, the specific implementation of step S22031-2 is as follows: using an optical flow method, such as the Lucas-Kanade optical flow algorithm, to track the position of the corner point in the first image, which is denoted as the second position of the corner point in the first image.

[0128] Step S22031-3: Determine the motion quantization value of the corner point based on the first position of the corner point in the second image and the second position of the corner point in the first image.

[0129] In one embodiment of this application, the specific implementation of the above steps S22031-3 can be as follows: for any corner point, calculate the displacement between the first position of the corner point in the second image and the second position in the first image; use the average displacement between the displacements corresponding to all corner points as the corner point motion quantization value; or determine the first score corresponding to the aforementioned determined average displacement according to the first preset mapping relationship, and record the first score as the corner point motion quantization value, wherein the first preset mapping is used to reflect the correspondence between different average displacements and corresponding scores, and is set according to experience.

[0130] It should be noted that in the embodiments shown in steps S22031-1 to S22031-3 above, the first image can essentially be the first image in the video stream, and the second image can essentially be the second image in the video stream. Of course, the first image can also essentially be the first grayscale image obtained based on step S22034 above, and the second image can essentially be the second grayscale image obtained based on step S22035 above.

[0131] Furthermore, the step of determining the motion region quantization value based on the first and second images in the video stream is implemented through the following steps S22031-4 to S22031-6.

[0132] Step S22031-4: Determine the differential image based on the first and second images in the video stream.

[0133] In this differential image, the pixel value of any pixel is the absolute value of the difference between the pixel values ​​of the corresponding pixel in the first image and the second image.

[0134] In this embodiment, the first image can essentially be the first image in the video stream, and the second image can essentially be the second image in the video stream. Of course, the first image can also essentially be the first grayscale image obtained based on the above step S22034, and the second image can essentially be the second grayscale image obtained based on the above step S22035.

[0135] Furthermore, for a pixel at position b in the first image, the pixel value at position b in the first image is subtracted from the pixel value at position b in the second image to obtain the difference value. The absolute value of the difference is then used as the pixel value at position b in the difference image. Based on this, the difference image can be obtained.

[0136] Step S22031-5: Perform binarization processing on the difference image according to the preset threshold to obtain the binarized image.

[0137] The preset threshold is the minimum change in pixel value caused by the movement of an object in the image. Based on experience or experimentation, this application does not limit the specific value of the preset threshold. It is understood that the movement of an object in the image includes: the position of the image acquisition device does not change, but objects in the environment captured by the image acquisition device move; or the position of the image acquisition device changes, but there are no moving objects in the environment captured by the image acquisition device; or the change in the position of the image acquisition device is asynchronous with the movement of objects in the environment captured by the image acquisition device, resulting in relative movement between the objects in the environment and the image acquisition device.

[0138] In this embodiment, the specific implementation of the above steps S22031-5 is as follows: for a pixel at position c in the difference image, if the pixel value of the pixel is greater than or equal to a preset threshold, the pixel value of the pixel at position c in the binarized image is determined to be 255; otherwise, the pixel value of the pixel at position c in the binarized image is determined to be 0, and a binarized image is obtained accordingly.

[0139] Step S22031-6: Determine the quantization value of the motion region based on the binarized image.

[0140] In one embodiment of this application, the number of pixels with a pixel value of 255 in the binarized image is used as the motion region quantization value. Alternatively, according to a second preset mapping relationship, a second score corresponding to the previously determined number of pixels with a pixel value of 255 is determined, and the second score is recorded as the motion region quantization value. The second preset mapping is used to reflect the correspondence between the number of pixels with different pixel values ​​of 255 and the corresponding score, and is set based on experience.

[0141] Step S22032: Determine the video scene type based on the corner motion quantization value and / or motion region quantization value.

[0142] In one embodiment of this application, the above step S22032 is specifically implemented by the following steps S22032-7, or S22032-8, or steps S22032-9 to S22032-11.

[0143] Step S22032-7: If the corner motion quantization value is greater than or equal to the preset corner motion change, determine that the video scene type is a dynamic scene.

[0144] The preset corner point motion change amount is the minimum change amount of the corner points in the image caused by the movement of objects in the image.

[0145] Corresponding to step S22032-7 above, if the corner motion quantization value is less than the preset corner motion change amount, the video scene type is determined to be a static scene.

[0146] Step S22032-8: If the motion region quantization value is greater than or equal to the preset motion region quantization value, determine that the video scene type is a dynamic scene.

[0147] The preset motion region quantization value is the minimum change in the motion region quantization value caused by the motion of an object in the image.

[0148] Corresponding to step S22032-8 above, if the motion region quantization value is less than the preset motion region quantization value, the video scene type is determined to be a static scene.

[0149] Step S22032-9: Perform normalization processing on the corner motion quantization value and the motion region quantization value respectively to obtain the normalized corner motion quantization value and the normalized motion region quantization value.

[0150] In this embodiment, a traditional normalization algorithm can be used to normalize the corner motion quantization value and the motion region quantization value respectively, so as to obtain the normalized corner motion quantization value and the normalized motion region quantization value.

[0151] Step S22032-10: Determine the motion quantization value based on the normalized corner motion quantization value and the normalized motion region quantization value.

[0152] In this embodiment, step S22032-10 is implemented by weighted summation. Specifically, the product of the first weight and the normalized corner motion quantization value is calculated and recorded as the first product; the product of the second weight and the normalized motion region quantization value is calculated and recorded as the second product; the sum of the first product and the second product is used as the motion quantization value; wherein, the sum of the first weight and the second weight is 1, and the values ​​of the first weight and the second weight are set according to experience.

[0153] Step S22032-11: If the motion quantization value is greater than or equal to the preset motion quantization value, determine that the video scene type is a dynamic scene.

[0154] The preset motion quantization value is the minimum change in motion quantization value caused by the movement of an object in the image.

[0155] Corresponding to step S22032-11, if the motion quantization value is less than the preset motion quantization value, the video scene type is determined to be a static scene.

[0156] Regarding the processing device 110 in the video acquisition system provided in this application, such as Figure 3 As shown, it is used to perform the following steps S310 to S340.

[0157] Step S310: Receive the video encoded stream and audio encoded stream sent by the head-mounted device 120.

[0158] Step S320: Decode the video encoded stream and the audio encoded stream respectively to obtain the video stream and the audio stream.

[0159] In this embodiment, based on the above steps S310 and S320, the video stream and audio stream from the first-person perspective collected by the head-mounted device 120 can be obtained.

[0160] Step S330: Perform image enhancement operation on the video stream to obtain an enhanced video stream.

[0161] In this embodiment, an image enhancement operation is performed on any frame of the video stream, and the video stream composed of the images obtained based on the image enhancement operation is denoted as the enhanced video stream. In one embodiment of this application, the image enhancement processing includes at least one of super-resolution, frame interpolation, text sharpening, and image quality enhancement processing.

[0162] Step S340 involves pushing the enhanced video and audio streams to a third-party live streaming platform.

[0163] In this embodiment, after receiving the enhanced data stream and audio stream sent by the processing device 110, the third-party live streaming platform distributes the enhanced data stream and audio stream to the client running the live streaming application. The client user can then watch the live stream from a first-person perspective.

[0164] Image enhancement and streaming operations are performed on the processing device 110 side, thus avoiding the head-mounted device 120 from performing image enhancement and streaming operations, thereby reducing the power consumption of the head-mounted device 120.

[0165] Based on the above, the video capture system provided in this application can perform live streaming through the head-mounted device 120 and the processing device 110. That is, the head-mounted device 120 can realize live streaming, which expands the functionality of the head-mounted device 120.

[0166] This application provides a video acquisition system, including: a head-mounted device for acquiring image data and audio data of a live-streaming subject, obtaining video and audio streams respectively, encoding the video and audio streams respectively, obtaining encoded video and encoded audio streams, and transmitting the encoded video and encoded audio streams to a processing device; and a processing device for receiving the encoded video and encoded audio streams sent by the head-mounted device, decoding the encoded video and encoded audio streams respectively, obtaining video and audio streams, performing image enhancement operations on the video streams to obtain enhanced video streams, and pushing the enhanced video and audio streams to a third-party live-streaming platform. Based on the video acquisition system provided by this application, the head-mounted device can realize live-streaming functionality, which expands the functionality of the head-mounted device.

[0167] In one embodiment of this application, the processing device 110 is further configured to perform the following step S350.

[0168] Step S350: Receive the live broadcast control command and send the live broadcast control command to the head-mounted device 120.

[0169] The live streaming control commands include any one of the following: start live streaming, stop live streaming, pause live streaming, and resume live streaming.

[0170] In this embodiment, the user inputs control commands to the processing device 110, such as triggering a "start" control to start the live broadcast in the display interface of the live broadcast application provided by the processing device 110.

[0171] Based on the above step S350, in the video acquisition system provided by this application, the head-mounted device 120 is also used to perform the following step S270.

[0172] Step S270: Receive live broadcast control instructions, and in response to the live broadcast control instructions, execute live broadcast control that matches the live broadcast control instructions.

[0173] Taking the live broadcast control command as the start live broadcast command as an example, after receiving the live broadcast start command, the head-mounted device 120 responds to the live broadcast start command by starting to collect image data and audio data in order to start the live broadcast.

[0174] Through the above steps S350 and S270, the video acquisition system provided in this application can also realize live broadcast control of the head-mounted device 120 by the processing device 110.

[0175] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0176] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0177] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0178] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0179] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0180] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0181] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0182] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0183] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A video acquisition system, characterized in that, include: A head-mounted device is used to collect image data and audio data of a live broadcast subject, obtain video streams and audio streams respectively, encode the video streams and audio streams respectively, obtain video encoded streams and audio encoded streams, and transmit the video encoded streams and audio encoded streams to a processing device; The processing device is configured to receive the video encoded stream and the audio encoded stream sent by the head-mounted device, decode the video encoded stream and the audio encoded stream respectively to obtain the video stream and the audio stream, perform image enhancement operation on the video stream to obtain the enhanced video stream, and push the enhanced video stream and the audio stream to a third-party live streaming platform.

2. The video acquisition system according to claim 1, characterized in that, The process of encoding the video stream and the audio stream respectively to obtain a video encoded stream and an audio encoded stream includes: Based on the first image and the second image in the video stream, the video scene type is determined. The video scene type includes static scenes and dynamic scenes. The first image is the frame preceding the second image. Based on the video scene type, the parameter values ​​of the encoding parameters of the video encoder are set to parameter values ​​that match the video scene type; The video encoder is controlled to encode the first image and the image of a preset number of subsequent frames to obtain a partial video encoded stream; The step of determining the video scene type based on the first and second images in the video stream is repeated, taking the next frame of a preset number of frames starting from the first image as the first image.

3. The video acquisition system according to claim 2, characterized in that, Determining the video scene type based on the first and second images in the video stream includes: Based on the first and second images in the video stream, determine the corner motion quantization value and / or motion region quantization value; The video scene type is determined based on the corner motion quantization value and / or motion region quantization value; Wherein, the corner motion quantization value is the quantization value of the motion vector formed by the corner point in the second image moving from the corresponding position in the first image to the corresponding position in the second image, and the motion region quantization value is the quantization value of the region size of the pixel-formed area that describes the motion of the second image compared to the first image.

4. The video acquisition system according to claim 3, characterized in that, Before determining the video scene type based on the first and second images in the video stream, the head-mounted device is further configured to: The first image and the second image in the video stream are respectively subjected to grayscale processing to obtain a first grayscale image corresponding to the first image and a second grayscale image corresponding to the second image; Update the first image to the first grayscale image; Update the second image to the second grayscale image.

5. The video acquisition system according to claim 3 or 4, characterized in that, The step of determining the corner motion quantization value and / or motion region quantization value based on the first image and the second image in the video stream includes: Based on the second image in the video stream, determine the first position of each corner point in the second image; For any of the corner points, determine the second position of the corner point in the first image from the first image; The motion quantization value of the corner point is determined based on the first position of the corner point in the second image and the second position of the corner point in the first image.

6. The video acquisition system according to claim 3 or 4, characterized in that, The step of determining the corner motion quantization value and / or motion region quantization value based on the first image and the second image in the video stream includes: A difference image is determined based on the first image and the second image in the video stream, wherein the pixel value of any pixel in the difference image is the absolute value of the difference between the pixel values ​​of the corresponding pixels in the first image and the second image; The difference image is binarized according to a preset threshold to obtain a binarized image. The preset threshold is the minimum change in pixel value caused by the movement of an object in the image. Based on the binarized image, determine the quantization value of the motion region.

7. The video acquisition system according to claim 3, characterized in that, The step of determining the video scene type based on the corner motion quantization value and / or motion region quantization value includes: If the corner motion quantization value is greater than or equal to the preset corner motion change amount, the video scene type is determined to be a dynamic scene; Alternatively, if the motion region quantization value is greater than or equal to the preset motion region quantization value, the video scene type is determined to be a dynamic scene; Alternatively, normalization processing can be performed on the corner motion quantization value and the motion region quantization value respectively to obtain normalized corner motion quantization value and normalized motion region quantization value; The motion quantization value is determined based on the normalized corner motion quantization value and the normalized motion region quantization value; If the motion quantization value is greater than or equal to the preset motion quantization value, the video scene type is determined to be a dynamic scene.

8. The video acquisition system according to claim 2, characterized in that, The encoding parameters include at least one of frame rate, bit rate, and GOP. The step of setting the parameter values ​​of the video encoder to match the video scene type includes: When the video scene type is a static scene, the frame rate of the video encoder is set to a first frame rate, the bit rate to a first bit rate, and the GOP to a first GOP; When the video scene type is a dynamic scene, the frame rate of the video encoder is set to a second frame rate, the bit rate is set to a second bit rate, and the GOP is set to a second GOP at least one of the following: Wherein, the first frame rate is less than the second frame rate, the first bitrate is less than the second bitrate, and the first GOP is greater than the second GOP.

9. The video acquisition system according to claim 1, characterized in that, The processing device is also used to receive live broadcast control instructions and send the live broadcast control instructions to the head-mounted device; The head-mounted device is also configured to receive the live streaming control command, and in response to the live streaming control command, execute live streaming control that matches the live streaming control command, the live streaming control command including any one of: start live streaming, stop live streaming, pause live streaming, and resume live streaming.

10. The video acquisition system according to claim 1, characterized in that, The step of transmitting the video encoded stream and audio encoded stream to the processing device includes: The video and audio encoded streams are transmitted to the processing device according to a preset traffic shaping strategy and / or a preset congestion control strategy.

11. The video acquisition system according to claim 1, characterized in that, The head-mounted device is also used to send data streams other than the video encoded stream and the audio encoded stream to the processing device; The step of transmitting the video encoded stream and the audio encoded stream to the processing device includes: when at least two of the data stream, the video encoded stream and the audio encoded stream need to be sent to the processing device at the same time, sending at least two of the data stream, the video encoded stream and the audio encoded stream according to a set priority; The priority is set such that the priority of the data stream is higher than the priority of the audio encoded stream, and the priority of the audio encoded stream is higher than the priority of the video encoded stream.

12. The video acquisition system according to claim 1, characterized in that, The head-mounted device is also used to store the video stream in a first memory and the audio stream in a second memory; The process of encoding the video stream and the audio stream respectively to obtain a video encoded stream and an audio encoded stream includes: The video encoder is controlled to read the video stream from the first memory and perform video encoding operations on the video stream to obtain a video encoded stream; The audio encoder is controlled to read the audio stream from the second memory, and an audio encoding operation is performed on the audio stream to obtain an audio encoded stream; The head-mounted device is also configured to read the video stream from the first memory and the audio stream from the second memory, and perform a preview operation on the read video stream and audio stream.

Images