A video stream encoding transmission method and device

Abstract

This application discloses a video stream encoding and transmission method and apparatus. The method includes: obtaining the door's open state; if the door's open state is closed, determining the dual-stream transmission mode as the current transmission mode; segmenting the image data of the original video into an original foreground region and an original background region; determining the current network quality index; determining the foreground frame rate of the original foreground region, the background frame rate of the original background region, and the background bitrate ratio based on the current network quality index; encoding the original foreground region and the original background region based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio to generate a dual-stream video; sending the dual-stream video to a streaming end; and the streaming end is used to fuse the dual-stream video. In this embodiment, by transmitting and fusing the background and foreground regions in a dual-stream manner, the technical effect of ensuring image quality while reducing bandwidth is achieved under network anomalies.

Description

Technical Field

[0001] This application relates to the field of video transmission technology, and in particular to a video stream encoding and transmission method and apparatus. Background Technology

[0002] With the improvement of people's living standards and the promotion and popularization of technologies such as the Internet, big data, artificial intelligence, and voice interaction, more and more traditional lifestyles are gradually changing, and the use of kitchen appliances is gradually moving towards intelligence. In order to monitor the cooking status inside the steam oven in real time, a visible light camera is integrated into the steam oven to capture cooking videos in real time during the cooking process. After encoding the cooking videos, they are transmitted to the user terminal, where users can view the cooking progress of the food inside the steam oven on the terminal's display screen in real time.

[0003] However, since the steam oven is located in the kitchen, which is prone to a weak network environment, if the amount of data transmitted in real time is too large, stuttering will occur during video viewing. Currently, real-time streaming mainly improves the smoothness of the video by adjusting the bit rate, frame rate, or resolution, but this will affect the display effect of the food to some extent. Summary of the Invention

[0004] To address the existing technical problems, this invention provides a video stream encoding and transmission method and apparatus. The background and foreground regions are encoded separately according to the door opening state and the current network state, and then transmitted in dual streams. The dual-stream video is then fused at the streaming end. By transmitting and fusing the background and foreground regions in dual streams, the technical effect of ensuring image quality while reducing bandwidth is achieved even under network anomalies.

[0005] In a first aspect, embodiments of this application provide a video stream encoding and transmission method applied to a server, the method comprising: Get the door's open / closed status; If the door is in the open state or closed state, the dual-stream transmission mode is determined as the current transmission mode; the dual-stream transmission mode indicates that the foreground and background areas of the video are transmitted separately at different frame rates. The image data of the original video is segmented into the original foreground region and the original background region; Determine the current network quality index; Determine the proportions of foreground frame rate, background frame rate, and background bitrate in the original foreground region based on the current network quality index. Based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio, the original foreground and original background regions are encoded to generate dual-stream video. The dual-stream video is sent to the pull stream end; the pull stream end is used to merge the dual-stream video.

[0006] In one optional embodiment, after obtaining the door's open state, the process includes: If the door is in the open state, then the single-stream transmission mode is determined as the current transmission mode. Single-stream transmission mode represents the transmission of video after the foreground and background regions are merged.

[0007] In one optional embodiment, the image data of the original video is segmented into an original foreground region and an original background region, including: Edge detection is performed on the image data to determine the bounding box of the image data; The first and second edge coordinates are determined based on the bounding box; the first and second edge coordinates are the coordinates of the diagonal points of the bounding box; the first and second edge coordinates are used to send to the pull stream end; The original foreground region and the original background region are determined based on the first edge coordinates and the second edge coordinates.

[0008] In one alternative embodiment, determining the current network quality index includes: Get the current bandwidth data, current packet loss rate, and current round-trip latency standard deviation; The current network quality index is determined based on the current bandwidth data, current packet loss rate, current round-trip delay standard deviation, and network quality index model.

[0009] In one optional embodiment, determining the foreground frame rate of the foreground region, the background frame rate of the background region, and the background bitrate ratio based on the current network quality index includes: If the current network quality index is less than or equal to the first threshold, the first frame rate is determined to be the background frame rate, the second frame rate is determined to be the foreground frame rate, and the first proportion is determined to be the background bitrate proportion; the second frame rate is greater than the first frame rate; or; If the current network quality index is greater than the first threshold but less than or equal to the second threshold, the third frame rate is determined to be the background frame rate, the fourth frame rate is determined to be the foreground frame rate, and the second percentage is determined to be the background bitrate percentage; the fourth frame rate is greater than the third frame rate; the third frame rate is greater than the first frame rate; the second percentage is less than the first percentage; or; If the current network quality index is greater than the second threshold, the fifth frame rate is determined to be the background frame rate, the sixth frame rate is determined to be the foreground frame rate, and the third percentage is determined to be the background bitrate percentage; the sixth frame rate is greater than the fifth frame rate; the fifth frame rate is greater than the third frame rate; and the third percentage is less than the second percentage.

[0010] Secondly, embodiments of this application provide a video stream encoding and transmission method applied at a streaming end, the method comprising: The system receives dual-stream video, first edge coordinates, and second edge coordinates from the server. The dual-stream video is generated by the server encoding the original foreground and background regions based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio. The current transmission mode is determined based on the door's open state. The foreground frame rate, background frame rate, and background bitrate ratio are determined based on the current network quality index. The original foreground and background regions are obtained by segmenting the image data from the original video. The dual-stream video includes foreground video and background video. The first and second edge coordinates are the diagonal coordinates of the bounding box of the image data. The foreground and background videos are fused based on the first and second edge coordinates.

[0011] In one optional embodiment, both the foreground video and the background video have timestamps; The foreground and background videos are fused based on the first and second edge coordinates, including: Generate virtual synchronization points; The virtual synchronization period is determined based on the foreground frame rate of the foreground video and the background frame rate of the background video. Network latency is determined based on the timestamps of the foreground and background videos; The time window is determined based on the virtual synchronization point, the virtual synchronization period, and network latency. Execute for each time window: If a foreground frame and background frame with matching timestamps exist within the time window, determine the foreground frame and background frame with matching timestamps as interpolated foreground frame and interpolated background frame; or; if no foreground frame and background frame with matching timestamps exist within the time window, determine the foreground frame and background frame based on the previous virtual synchronization cycle as interpolated foreground frame and interpolated background frame. Decode the interpolated foreground and background frames into a three-primary-color format; Based on the first edge coordinates and the second edge coordinates, the interpolated foreground frame is interpolated into the interpolated background frame.

[0012] Thirdly, embodiments of this application provide a video stream encoding and transmission apparatus, applied to a server, the apparatus comprising: The acquisition module is used to obtain the door's open / closed status. The first determining module is used to determine the dual-stream transmission mode as the current transmission mode if the door is in the open state or closed state; the dual-stream transmission mode represents that the foreground and background areas of the video are transmitted separately at different frame rates. The region segmentation module is used to segment the image data of the original video into the original foreground region and the original background region; The second determining module is used to determine the current network quality index; The third determining module is used to determine the proportion of the foreground frame rate of the original foreground region, the background frame rate of the original background region, and the background bitrate based on the current network quality index. The encoding module is used to encode the original foreground and background regions based on the current transmission mode, foreground frame rate, background frame rate, and background bit rate ratio to generate dual-stream video; The sending module is used to send the dual-stream video to the pulling end; the pulling end is used to merge the dual-stream video.

[0013] Fourthly, embodiments of this application provide a video stream encoding and transmission apparatus, applied at a streaming end, the apparatus comprising: The receiving module is used to receive dual-stream video, first edge coordinates, and second edge coordinates sent by the server. The dual-stream video is generated by the server encoding the original foreground and original background regions based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio. The current transmission mode is determined based on the door opening state. The foreground frame rate, background frame rate, and background bitrate ratio are determined based on the current network quality index. The original foreground and original background regions are obtained by segmenting the image data of the original video. The dual-stream video includes foreground video and background video. The first edge coordinates and second edge coordinates are the diagonal coordinates of the bounding box of the image data. The fusion module is used to fuse foreground and background videos based on first and second edge coordinates.

[0014] Fifthly, embodiments of this application provide an intelligent kitchen appliance, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the video stream encoding and transmission method of the first or second aspect.

[0015] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the video stream encoding and transmission method of the first or second aspect.

[0016] In a seventh aspect, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video stream encoding and transmission method of the first or second aspect.

[0017] The video stream encoding and transmission method and apparatus provided in this application have the following technical effects: The process involves: obtaining the door's open state; if the door is closed, determining the dual-stream transmission mode as the current transmission mode; the dual-stream transmission mode represents transmitting the foreground and background regions of the video at different frame rates; segmenting the image data of the original video into the original foreground and original background regions; determining the current network quality index; determining the foreground frame rate of the original foreground region, the background frame rate of the original background region, and the background bitrate ratio based on the current network quality index; encoding the original foreground and original background regions based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio to generate a dual-stream video; sending the dual-stream video to the streaming end; and the streaming end fusing the dual-stream video. In this embodiment, the background and foreground regions are encoded separately according to the door's open state and the current network state for dual-stream transmission, and then fused at the streaming end. By transmitting and fusing the background and foreground regions simultaneously, the technical effect of maintaining image quality while reducing bandwidth is achieved even under network anomalies. Attached Figure Description

[0018] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a video stream encoding and transmission method provided in an embodiment of this application. Figure 1 ; Figure 3 This is a flowchart illustrating a video stream encoding and transmission method provided in an embodiment of this application. Figure 2 ; Figure 4 This is a flowchart illustrating a video stream encoding and transmission method provided in an embodiment of this application. Figure 3 ; Figure 5 This is a schematic diagram of the structure of a video stream encoding and transmission device provided in an embodiment of this application. Figure 1 ; Figure 6 This is a schematic diagram of the structure of a video stream encoding and transmission device provided in an embodiment of this application. Figure 2 ; Figure 7 This is a hardware structure block diagram of a server for a video stream encoding and transmission method provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application. The steam oven 100 includes a pressure sensor 101, an image acquisition module 102, a server 103, and a flow-drawing end 104, and also includes a user terminal 200.

[0023] In one possible embodiment, the steam oven 100 is a possible cooking device; in other embodiments, it may also be other cooking devices with the function of transmitting real-time video of the food cooking inside.

[0024] In one possible embodiment, a pressure sensor 101 is installed on the door of the steam oven 100 to detect the door's open state and send the door's open state to the server 103.

[0025] In one possible embodiment, the image acquisition module 102 is specifically configured as a visible light camera, which is installed inside the steam oven 100 to acquire raw video of the food cooking inside in real time and send the raw video to the server 103.

[0026] In one possible embodiment, server 103 obtains the door's open state; if the door's open state is closed, it determines that the dual-stream transmission mode is the current transmission mode; the dual-stream transmission mode represents transmitting the foreground and background regions of the video separately at different frame rates; the image data of the original video is segmented into the original foreground region and the original background region; the current network quality index is determined; based on the current network quality index, the foreground frame rate of the original foreground region, the background frame rate of the original background region, and the background bitrate ratio are determined; based on the current transmission mode, the foreground frame rate, the background frame rate, and the background bitrate ratio, the original foreground region and the original background region are encoded to generate a dual-stream video; the dual-stream video is sent to the streaming end; the streaming end is used to fuse the dual-stream video.

[0027] In one possible embodiment, the streaming end 104 receives dual-stream video, first edge coordinates, and second edge coordinates sent by the server; the dual-stream video is generated by the server encoding the original foreground region and the original background region based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio; the current transmission mode is determined based on the gate opening state; the foreground frame rate, background frame rate, and background bitrate ratio are determined based on the current network quality index; the original foreground region and the original background region are obtained by segmenting the image data of the original video; the dual-stream video includes foreground video and background video; the first edge coordinates and the second edge coordinates are the diagonal coordinates of the bounding box of the image data; the foreground video and the background video are fused based on the first edge coordinates and the second edge coordinates.

[0028] In one possible embodiment, the user terminal 200 can be a mobile phone, computer, smart wearable device, etc., for receiving the cooking video fused by the streaming end 104.

[0029] In this embodiment, the background and foreground regions are encoded separately according to the door opening state and the current network state, and then transmitted in dual streams. The dual-stream video is then fused at the streaming end. By transmitting and fusing the background and foreground regions in dual streams, the technical effect of ensuring image quality while reducing bandwidth is achieved even in the event of network anomalies.

[0030] The following describes a specific embodiment of a video stream encoding and transmission method according to this application. Figure 2 This is a flowchart illustrating a video stream encoding and transmission method provided in an embodiment of this application. Figure 1 This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the embodiments or drawings... Figure 2 As shown, this method, applied to a server, may include: S201: Obtain the door opening status.

[0031] S202: If the door is in the closed state, determine the dual-stream transmission mode as the current transmission mode; the dual-stream transmission mode indicates that the foreground and background areas of the video are transmitted separately at different frame rates.

[0032] S203: Segment the image data of the original video into the original foreground region and the original background region.

[0033] S204: Determine the current network quality index.

[0034] S205: Determine the proportion of foreground frame rate, background frame rate, and background bitrate of the original foreground region based on the current network quality index.

[0035] S206: Encode the original foreground and background regions based on the current transmission mode, foreground frame rate, background frame rate, and background bit rate ratio to generate dual-stream video.

[0036] S207: Send the dual-stream video to the pull stream end; the pull stream end is used to merge the dual-stream video.

[0037] Figure 3 This is a flowchart illustrating a video stream encoding and transmission method provided in an embodiment of this application. The method may include: S301: Obtain the door opening status.

[0038] In this embodiment, when the door is closed, the door will press against the pressure sensor on the door, so the opening status of the door can be detected by the pressure sensor installed on the door of the steam oven.

[0039] S302: Determine if the door is closed. If yes, execute S303; otherwise, execute S304.

[0040] S303: Determine the dual-stream transmission mode as the current transmission mode.

[0041] In this embodiment, the dual-stream transmission mode represents the separate transmission of the foreground and background regions of the video at different frame rates. In a cooking video, the foreground region specifically refers to the food ingredients, which undergo changes in color and volume as the cooking process progresses, while the background region may be a baking pan, etc., which does not change significantly during the cooking process.

[0042] S304: Determine single-stream transmission mode as the current transmission mode.

[0043] In the embodiments of this application, the single-stream transmission mode represents the transmission of a video after fusing the foreground and background regions.

[0044] In one possible embodiment, when the door is detected to be closed and the oven is in the cooking process, the background and foreground areas can be transmitted separately at different frame rates. However, when the door is open, the user may continue to add ingredients or perform other operations. If the background area has a lower frame rate during user operations, it will affect the user experience. Therefore, the system switches to a single-stream transmission mode, and the background area and the food area are broadcast live as a single frame.

[0045] S305: Segment the image data of the original video into the original foreground region and the original background region.

[0046] In one optional embodiment, the image data of the original video is segmented into an original foreground region and an original background region, including: S3051: Perform edge detection on the image data to determine the bounding box of the image data.

[0047] In this embodiment of the application, the bounding box is the smallest rectangular bounding box that can enclose the food in the image.

[0048] S3052: Determine the coordinates of the first edge and the second edge based on the bounding box.

[0049] In this embodiment, the first edge coordinates and the second edge coordinates are the coordinates of the diagonal points of the bounding box, specifically the coordinates of the top left corner. , ) and the coordinates of the lower right corner ( , The first and second edge coordinates are used to send to the streaming end, which can re-merge the segmented foreground and background regions based on the first and second edge coordinates.

[0050] S3053: Determine the original foreground region and the original background region based on the first edge coordinates and the second edge coordinates.

[0051] S306: Determine the current network quality index.

[0052] In one alternative embodiment, determining the current network quality index includes: S3061: Get the current bandwidth data, current packet loss rate, and current round-trip time standard deviation.

[0053] S3062: Determine the current network quality index based on the current bandwidth data, current packet loss rate, current round-trip delay standard deviation, and network quality index model.

[0054] In this embodiment of the application, the network quality index model is as follows: Where NQI is the network quality index, and B is the current bandwidth. To preset the maximum bandwidth, The standard deviation of Round-Trip Time (RTT) For time delay sensitivity coefficient, =0.1, where L is the packet loss rate and the weighting coefficient is 0.1. , , Therefore, the current network quality index can be determined by detecting the current bandwidth data, the current packet loss rate, and the current round-trip time standard deviation, thereby enabling real-time monitoring of network quality.

[0055] S307: Determine the proportion of foreground frame rate, background frame rate, and background bitrate of the original foreground region based on the current network quality index.

[0056] In this application, the current frame rate is set according to a frame rate grading strategy, and the bitrate is adjusted according to a bitrate control algorithm, specifically using a constant bitrate for the background region. The foreground region uses a dynamic bitrate, with a bitrate of [value missing]. , This represents the percentage of background bitrate.

[0057] In one optional embodiment, determining the foreground frame rate of the foreground region, the background frame rate of the background region, and the background bitrate ratio based on the current network quality index includes: S3071: If the current network quality index is less than or equal to the first threshold, determine the first frame rate as the background frame rate, determine the second frame rate as the foreground frame rate, and determine the first proportion as the background bitrate proportion.

[0058] In one possible embodiment, the second frame rate is greater than the first frame rate, and the first threshold is 0.5. If network congestion is detected, the background frame rate for the background area is set to 1fps, and the foreground frame rate for the food area is set to 15fps. The background bitrate percentage is then set to... .

[0059] S3072: If the current network quality index is greater than the first threshold but less than or equal to the second threshold, determine the third frame rate as the background frame rate, determine the fourth frame rate as the foreground frame rate, and determine the second proportion as the background bitrate proportion.

[0060] In one possible embodiment, the fourth frame rate is greater than the third frame rate; the third frame rate is greater than the first frame rate; and the second percentage is less than the first percentage.

[0061] In one possible embodiment, the second threshold is 0.8, if The current network is judged to be average, with the background area having a frame rate of 5fps and the foreground area of ​​the food having a frame rate of 25fps. The background bitrate percentage is then set to... .

[0062] S3073: If the current network quality index is greater than the second threshold, determine the fifth frame rate as the background frame rate, determine the sixth frame rate as the foreground frame rate, and determine the third percentage as the background bitrate percentage.

[0063] In one possible embodiment, the sixth frame rate is greater than the fifth frame rate; the fifth frame rate is greater than the third frame rate; and the third percentage is less than the second percentage.

[0064] In one possible embodiment, if The network is then determined to be unobstructed, with the background area at 15fps and the food area at 30fps. The background bitrate percentage is then set to... =20%.

[0065] S308: Encodes the original foreground and background regions based on the current transmission mode, foreground frame rate, background frame rate, and background bit rate ratio to generate dual-stream video.

[0066] S309: Send dual-stream video to the streaming end.

[0067] In this embodiment, the first and second edge coordinates of the top and bottom corners of the food are output to the streaming end before the live broadcast. A timestamp is added to each frame of the image, and the images are transmitted in real time to a Real-Time Massaging Protocol (RTMP) server or live broadcast in real time via Real-Time Streaming Protocol (RTSP). The streaming end is used to fuse the two streams of video.

[0068] Figure 4 This is a flowchart illustrating a video stream encoding and transmission method provided in an embodiment of this application. Figure 3 Applied to the pull end, the method may include: S401: Receive dual-stream video, first edge coordinates, and second edge coordinates sent by the server.

[0069] In one possible embodiment, the streaming end pulls the stream from the server in real time via RTMP, or receives dual-stream video, first edge coordinates, and second edge coordinates in real time via RTSP.

[0070] In this embodiment, the dual-stream video is generated by the server encoding the original foreground region and the original background region based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio. The current transmission mode is determined based on the gate's open state, the foreground frame rate, background frame rate, and background bitrate ratio are determined based on the current network quality index, and the original foreground region and original background region are obtained by segmenting the image data of the original video.

[0071] Dual-stream video includes foreground video and background video, and the first edge coordinates and the second edge coordinates are the coordinates of the diagonal points of the bounding box of the image data.

[0072] S402: Fuse foreground and background videos based on first and second edge coordinates.

[0073] In this embodiment, both the foreground video and the background video have timestamps.

[0074] In one optional embodiment, fusing the foreground video and the background video based on a first edge coordinate and a second edge coordinate includes: S4021: Generate a virtual synchronization point.

[0075] S4022: Determine the virtual synchronization period based on the foreground frame rate of the foreground video and the background frame rate of the background video.

[0076] S4023: Determine network latency based on the timestamps of the foreground and background videos.

[0077] S4024: Determine the time window based on the virtual synchronization point, virtual synchronization period, and network latency.

[0078] In this embodiment of the application, a virtual synchronization point is generated. Virtual synchronization period for Network latency is calculated using the send and receive timestamps within data packets. Finally, the time window was obtained. T / 2.

[0079] Execute for each time window: S4025: Determine if there are foreground and background frames with matching timestamps within the time window. If yes, execute S4026; otherwise, execute S4027.

[0080] In this embodiment, the foreground and background frames with matching timestamps are the dual-stream frames with the closest timestamps within the time window.

[0081] S4026: Determine the foreground and background frames with matching timestamps as interpolated foreground and background frames.

[0082] S4027: The foreground and background frames based on the previous virtual synchronization cycle are interpolated foreground and background frames.

[0083] In one possible embodiment, if a foreground frame and a background frame with matching timestamps exist within the time window, the foreground frame and background frame with matching timestamps are determined to be interpolated foreground frames and interpolated background frames.

[0084] In another possible embodiment, if there are no timestamp-matching foreground and background frames within the time window, the foreground and background frames based on the previous virtual synchronization cycle are interpolated foreground and background frames.

[0085] Within the time window, select the dual-stream frame with the closest timestamp. If there is no corresponding frame in the matching window, use the frame from the previous cycle for interpolation to avoid interruption of the synthesis due to frame loss in the single stream and maintain the continuity of the image.

[0086] S4028: Decode the interpolated foreground and background frames into a three-color format.

[0087] The interpolated foreground and background frames to be merged are decoded into a three-primary-color format, namely RGB format, to facilitate subsequent image interpolation processing.

[0088] S4029: Based on the first edge coordinates and the second edge coordinates, interpolate the foreground frame to the background frame.

[0089] In this embodiment of the application, based on the obtained upper left corner coordinates ( , ) and the coordinates of the lower right corner ( , ), interpolate the (0,0) to (w,h) pixels of the food region to the background region ( , (), , This allows for the fusion of the two areas, thereby generating a complete cooking image.

[0090] Where w is the horizontal resolution of the food region and h is the vertical resolution of the food region.

[0091] This application also provides a video stream encoding and transmission apparatus. Figure 5 This is a schematic diagram of the structure of a video stream encoding and transmission device provided in an embodiment of this application. Figure 1 ,like Figure 5 As shown, the device 500, applied to a server, includes: Module 501 is used to obtain the door opening status; The first determining module 502 is used to determine the dual-stream transmission mode as the current transmission mode if the door is in the open state or closed state; the dual-stream transmission mode represents that the foreground area and background area of ​​the video are transmitted separately at different frame rates. The region segmentation module 503 is used to segment the image data of the original video into an original foreground region and an original background region; The second determining module 504 is used to determine the current network quality index; The third determining module 505 is used to determine the proportion of the foreground frame rate of the original foreground region, the background frame rate of the original background region, and the background bitrate based on the current network quality index. The encoding module 506 is used to encode the original foreground region and the original background region based on the current transmission mode, foreground frame rate, background frame rate and background bit rate ratio to generate dual-stream video; The sending module 507 is used to send the dual-stream video to the pulling end; the pulling end is used to merge the dual-stream video.

[0092] In an optional embodiment, it further includes: The fourth determining module is used to determine the single-stream transmission mode as the current transmission mode if the door is in the open state. Single-stream transmission mode represents the transmission of video after the foreground and background regions are merged.

[0093] In an optional embodiment, it further includes: The fifth determination module is used to perform edge detection on the image data and determine the bounding box of the image data; The sixth determining module is used to determine the first edge coordinates and the second edge coordinates based on the bounding box; the first edge coordinates and the second edge coordinates are the coordinates of the diagonal points of the bounding box; the first edge coordinates and the second edge coordinates are used to send to the pull stream end; The seventh determination module is used to determine the original foreground region and the original background region based on the first edge coordinates and the second edge coordinates.

[0094] In an optional embodiment, it further includes: The first acquisition module is used to acquire current bandwidth data, current packet loss rate, and current round-trip latency standard deviation. The eighth determination module is used to determine the current network quality index based on the current bandwidth data, the current packet loss rate, the current round-trip delay standard deviation, and the network quality index model.

[0095] In an optional embodiment, it further includes: The ninth determining module is used to determine the first frame rate as the background frame rate, the second frame rate as the foreground frame rate, and the first proportion as the background bitrate proportion if the current network quality index is less than or equal to the first threshold; or; The tenth determining module is used to determine the third frame rate as the background frame rate, the fourth frame rate as the foreground frame rate, and the second proportion as the background bitrate proportion if the current network quality index is greater than the first threshold but less than or equal to the second threshold; the fourth frame rate is greater than the third frame rate; the third frame rate is greater than the first frame rate; the second proportion is less than the first proportion; or; The eleventh determination module is used to determine the fifth frame rate as the background frame rate, the sixth frame rate as the foreground frame rate, and the third percentage as the background bitrate percentage if the current network quality index is greater than the second threshold; the sixth frame rate is greater than the fifth frame rate; the fifth frame rate is greater than the third frame rate; and the third percentage is less than the second percentage.

[0096] This application also provides a video stream encoding and transmission apparatus. Figure 6 This is a schematic diagram of the structure of a video stream encoding and transmission device provided in an embodiment of this application. Figure 2 ,like Figure 6 As shown, the device 600, applied to the drawing end, includes: The receiving module 601 is used to receive dual-stream video, first edge coordinates, and second edge coordinates sent by the server. The dual-stream video is generated by the server encoding the original foreground region and the original background region based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio. The current transmission mode is determined based on the door opening state. The foreground frame rate, background frame rate, and background bitrate ratio are determined based on the current network quality index. The original foreground region and the original background region are obtained by segmenting the image data of the original video. The dual-stream video includes foreground video and background video. The first edge coordinates and the second edge coordinates are the diagonal coordinates of the bounding box of the image data. The fusion module 602 is used to fuse the foreground video and the background video based on the first edge coordinates and the second edge coordinates.

[0097] In one optional embodiment, both the foreground video and the background video have timestamps; further comprising: The generation module is used to generate virtual synchronization points; The twelfth determination module is used to determine the virtual synchronization period based on the foreground frame rate of the foreground video and the background frame rate of the background video. The thirteenth determination module is used to determine network latency based on the timestamps of the foreground and background videos; The fourteenth determination module is used to determine the time window based on the virtual synchronization point, the virtual synchronization period, and the network latency. Execute for each time window: The fifteenth determining module is used to determine the foreground and background frames with matching timestamps as interpolated foreground and background frames if there are foreground and background frames with matching timestamps within the time window; or if there are no foreground and background frames with matching timestamps within the time window, the foreground and background frames based on the previous virtual synchronization period are interpolated foreground and background frames. The decoding module is used to decode the interpolated foreground and background frames into a three-primary-color format; The interpolation module is used to interpolate the foreground frame to the background frame based on the first edge coordinates and the second edge coordinates.

[0098] The apparatus and method embodiments in this application are based on the same application concept.

[0099] The methods and embodiments provided in this application can be executed on a computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 7 This is a hardware structure block diagram of a server for a video stream encoding and transmission method provided in an embodiment of this application. For example... Figure 7 As shown, the server 700 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 710 (CPUs 710 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 730 for storing data, and one or more storage media 720 (e.g., one or more mass storage devices) for storing application programs 723 or data 722. The memory 730 and storage media 720 may be temporary or persistent storage. The program stored in the storage media 720 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 710 may be configured to communicate with the storage media 720 and execute the series of instruction operations stored in the storage media 720 on the server 700. Server 700 may also include one or more power supplies 760, one or more wired or wireless network interfaces 750, one or more input / output interfaces 740, and / or one or more operating systems 721, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0100] The input / output interface 740 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 700. In one example, the input / output interface 740 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 740 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0101] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 700 may also include... Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown.

[0102] This application provides an intelligent kitchen appliance, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the above-described data processing method.

[0103] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in a server to store at least one instruction, at least one program, code set, or instruction set related to implementing a video stream encoding and transmission method in the method embodiment. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the video stream encoding and transmission method described above.

[0104] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0105] As can be seen from the embodiments of the video stream encoding and transmission method, apparatus, smart kitchen appliance, or storage medium provided in this application, the present application obtains the door opening state; if the door opening state is closed, the dual-stream transmission mode is determined as the current transmission mode; the dual-stream transmission mode represents transmitting the foreground and background regions of the video separately at different frame rates; the image data of the original video is segmented into the original foreground region and the original background region; the current network quality index is determined; based on the current network quality index, the foreground frame rate of the original foreground region, the background frame rate of the original background region, and the background bitrate ratio are determined; based on the current transmission mode, the foreground frame rate, the background frame rate, and the background bitrate ratio, the original foreground region and the original background region are encoded to generate a dual-stream video; the dual-stream video is sent to the streaming end; the streaming end is used to fuse the dual-stream video. In the embodiments of this application, the background region and the foreground region are encoded separately according to the door opening state and the current network state, and dual-stream transmission is performed. The dual-stream video is then fused at the streaming end. By transmitting and fusing the background and foreground regions in a dual-stream manner, the technical effect of ensuring image quality while reducing bandwidth is achieved under network abnormalities.

[0106] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0107] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0108] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0109] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A video stream encoding and transmission method, characterized in that, Applied to servers, including: Get the door's open / closed status; If the door is in a closed state, the dual-stream transmission mode is determined to be the current transmission mode; the dual-stream transmission mode indicates that the foreground and background areas of the video are transmitted separately at different frame rates. The image data of the original video is segmented into the original foreground region and the original background region; Determine the current network quality index; The foreground frame rate of the original foreground region, the background frame rate of the original background region, and the background bitrate ratio are determined based on the current network quality index. The original foreground region and the original background region are encoded based on the current transmission mode, the foreground frame rate, the background frame rate, and the background bitrate ratio to generate dual-stream video; The dual-stream video is sent to the streaming end; the streaming end is used to merge the dual-stream video.

2. The video stream encoding and transmission method according to claim 1, characterized in that, After obtaining the door's open state, the following steps are included: If the door is in the open state, then the single-stream transmission mode is determined to be the current transmission mode. The single-stream transmission mode represents the transmission of the video after fusing the foreground and background regions.

3. The video stream encoding and transmission method according to claim 1, characterized in that, The step of segmenting the image data of the original video into the original foreground region and the original background region includes: Edge detection is performed on the image data to determine the bounding box of the image data; The first edge coordinates and the second edge coordinates are determined based on the bounding box; the first edge coordinates and the second edge coordinates are the coordinates of the diagonal points of the bounding box; the first edge coordinates and the second edge coordinates are used to send to the pull end; The original foreground region and the original background region are determined based on the first edge coordinates and the second edge coordinates.

4. The video stream encoding and transmission method according to claim 1, characterized in that, Determining the current network quality index includes: Get the current bandwidth data, current packet loss rate, and current round-trip latency standard deviation; The current network quality index is determined based on the current bandwidth data, the current packet loss rate, the current round-trip delay standard deviation, and the network quality index model.

5. A video stream encoding and transmission method according to claim 4, characterized in that, The step of determining the foreground frame rate of the foreground region, the background frame rate of the background region, and the background bitrate ratio based on the current network quality index includes: If the current network quality index is less than or equal to a first threshold, the first frame rate is determined to be the background frame rate, the second frame rate is determined to be the foreground frame rate, and the first proportion is determined to be the background bitrate proportion; the second frame rate is greater than the first frame rate; or; If the current network quality index is greater than the first threshold but less than or equal to the second threshold, the third frame rate is determined to be the background frame rate, the fourth frame rate is determined to be the foreground frame rate, and the second percentage is determined to be the background bitrate percentage; the fourth frame rate is greater than the third frame rate; the third frame rate is greater than the first frame rate; the second percentage is less than the first percentage; or; If the current network quality index is greater than the second threshold, the fifth frame rate is determined to be the background frame rate, the sixth frame rate is determined to be the foreground frame rate, and the third proportion is determined to be the background bitrate proportion; the sixth frame rate is greater than the fifth frame rate; the fifth frame rate is greater than the third frame rate; and the third proportion is less than the second proportion.

6. A video stream encoding and transmission method, characterized in that, Applied to the pull end, including: The system receives dual-stream video, first edge coordinates, and second edge coordinates sent by a server. The dual-stream video is generated by the server encoding the original foreground and original background regions based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio. The current transmission mode is determined based on the door's open state. The foreground frame rate, background frame rate, and background bitrate ratio are determined based on the current network quality index. The original foreground and original background regions are obtained by segmenting the image data of the original video. The dual-stream video includes foreground video and background video. The first edge coordinates and second edge coordinates are the diagonal coordinates of the bounding box of the image data. The foreground video and the background video are fused based on the first edge coordinates and the second edge coordinates.

7. A video stream encoding and transmission method according to claim 6, characterized in that, Both the foreground video and the background video have timestamps; The process of fusing the foreground video and the background video based on the first edge coordinates and the second edge coordinates includes: Generate virtual synchronization points; The virtual synchronization period is determined based on the foreground frame rate of the foreground video and the background frame rate of the background video; Network latency is determined based on the timestamps of the foreground video and the background video; A time window is determined based on the virtual synchronization point, the virtual synchronization period, and the network latency. For each of the aforementioned time windows, execute: If a foreground frame and a background frame with matching timestamps exist within the time window, the foreground frame and background frame with matching timestamps are determined to be interpolated foreground frames and interpolated background frames; or; if no foreground frame and background frame with matching timestamps exist within the time window, the foreground frame and background frame based on the previous virtual synchronization cycle are the interpolated foreground frames and the interpolated background frames. Decode the interpolated foreground frame and the interpolated background frame into a three-primary-color format; Based on the first edge coordinates and the second edge coordinates, the interpolated foreground frame is interpolated into the interpolated background frame.

8. A video stream encoding and transmission apparatus, characterized in that, Applied to servers, including: The acquisition module is used to obtain the door's open / closed status. The first determining module is used to determine the dual-stream transmission mode as the current transmission mode if the door is in a closed state; the dual-stream transmission mode represents transmitting the foreground and background regions of the video separately at different frame rates. The region segmentation module is used to segment the image data of the original video into the original foreground region and the original background region; The second determining module is used to determine the current network quality index; The third determining module is used to determine the foreground frame rate of the original foreground region, the background frame rate of the original background region, and the background bitrate ratio based on the current network quality index. The encoding module is used to encode the original foreground region and the original background region based on the current transmission mode, the foreground frame rate, the background frame rate, and the background bitrate ratio to generate dual-stream video; The sending module is used to send the dual-stream video to the pulling end; the pulling end is used to fuse the dual-stream video.

9. A video stream encoding and transmission apparatus, characterized in that, Applied to the pull end, including: A receiving module is used to receive dual-stream video, first edge coordinates, and second edge coordinates sent by a server. The dual-stream video is generated by the server encoding the original foreground region and the original background region based on the current transmission mode, foreground frame rate, background frame rate, and background bitrate ratio. The current transmission mode is determined based on the door opening state. The foreground frame rate, background frame rate, and background bitrate ratio are determined based on the current network quality index. The original foreground region and the original background region are obtained by segmenting the image data of the original video. The dual-stream video includes foreground video and background video. The first edge coordinates and the second edge coordinates are the diagonal coordinates of the bounding box of the image data. A fusion module is used to fuse the foreground video and the background video based on the first edge coordinates and the second edge coordinates.

Images