MJPEG-based single-channel multi-stream video transmission device and method

Abstract

This invention discloses a single-channel multi-channel video stream transmission device and method based on MJPEG. The device includes a camera processing module and a rear locking plate. The camera processing module includes multiple encoders of different video formats and multiplexing modules, with the encoders of different video formats connected to the multiplexing modules respectively. The rear locking plate is connected to the camera processing module to communicate with the multiplexing modules in the camera processing module via the UVC protocol. Thus, when the rear locking plate only supports one MJPEG transmission, adding multiple encoders of different video formats and multiplexing modules to the camera processing module allows for multiplexing of multiple video streams for transmission. This enables the rear locking plate to select the appropriate format and resolution video stream for forwarding to the corresponding device based on network conditions, thereby reducing hardware and development costs.

Description

Technical Field

[0001] This invention relates to the field of video transmission technology, and in particular to a single-channel multi-channel video stream transmission device and a single-channel multi-channel video stream transmission method based on MJPEG. Background Technology

[0002] In related technologies, smart door locks typically include functions such as fingerprint recognition, rear screen display, remote real-time audio and video intercom, and facial recognition. Traditional lock manufacturers hope to upgrade their products at a lower cost based on existing solutions. However, due to limitations in the processing capabilities of the rear lock board chip, which only supports MJPEG decoding and cannot support formats such as H264 or H265, and cannot perform multi-channel video encoding and decoding or image scaling, upgrading the rear lock board or placing the video stream forwarding function in the camera processing module would result in high hardware and development costs. Summary of the Invention

[0003] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. Therefore, one objective of this invention is to provide a single-channel multi-stream video transmission device based on MJPEG, which, when only one MJPEG transmission is supported, multiple video streams are multiplexed for transmission at the front end; the back end selects appropriate format and resolution video streams for forwarding to the corresponding devices based on network conditions, thereby reducing hardware and development costs.

[0004] The second objective of this invention is to propose a single-channel multi-stream video stream transmission method based on MJPEG.

[0005] To achieve the above objectives, the first aspect of this invention proposes a single-channel multi-channel video stream transmission device based on MJPEG, comprising: a camera processing module, the camera processing module including multiple encoders of different video formats and a multi-channel encapsulation module, the multiple encoders of different video formats being respectively connected to the multi-channel encapsulation module; and a rear locking plate, the rear locking plate being connected to the camera processing module to communicate with the multi-channel encapsulation module in the camera processing module via the UVC protocol.

[0006] The single-channel multi-stream video stream transmission device based on MJPEG proposed in this invention includes multiple encoders and multi-encapsulation modules for different video formats on the camera processing module, with each encoder connected to a multi-encapsulation module via the UVC protocol. The rear locking plate is connected to the camera processing module to communicate with the multi-encapsulation module via the UVC protocol. Thus, even when the rear locking plate only supports one MJPEG transmission, adding multiple encoders and multi-encapsulation modules for different video formats to the camera processing module allows for multiplexing of multiple video streams for transmission. This enables the rear locking plate to select the appropriate format and resolution video stream for forwarding to the corresponding device based on network conditions, thereby reducing hardware and development costs.

[0007] In addition, the single-channel multi-stream video transmission device based on MJPEG proposed above according to the present invention may also have the following additional technical features:

[0008] Optionally, the encoders for the multiple different video formats include an MJPEG encoder, an H264 encoder, and an H265 encoder.

[0009] Optionally, the MJPEG encoder is used to encode the received raw image to obtain an MJPEG image, the H264 encoder is used to encode the received raw image to obtain an H264 image, and the H265 encoder is used to encode the received raw image to obtain an H265 image.

[0010] Optionally, the multiplexing module is used to receive the MJPEG image, the H264 image, and the H265 image, and encapsulate the H264 image in a pseudo-MJPEG image and the H265 image in a pseudo-MJPEG image.

[0011] Optionally, a private protocol is added to the extended tag segment in the tag code of the MJPEG image or pseudo-MJPEG image to encapsulate the images of different video formats in the extended tag segment.

[0012] Optionally, the rear locking plate is used to receive the MJPEG image and the pseudo-MJPEG image in order to obtain the corresponding MJPEG video stream, H264 video stream and H265 video stream.

[0013] Optionally, the MJPEG-based single-channel multi-channel video stream transmission device further includes a cloud platform, a first display device, and a second display device. The cloud platform, the first display device, and the second display device are respectively connected to the rear locking plate so that the rear locking plate transmits the received H264 video stream to the cloud platform, transmits the received MJPEG video stream to the first display device, and transmits the received H265 video stream to the second display device.

[0014] To achieve the above objectives, a second aspect of the present invention proposes a single-channel multi-channel video stream transmission method based on MJPEG, comprising the following steps: multiple encoders of different video formats disposed in the camera processing module encode the received raw images to obtain corresponding images of different video formats; a multi-channel encapsulation module disposed in the camera processing module receives the images of different video formats and encapsulates the images of different video formats in corresponding MJPEG images or pseudo-MJPEG images; a rear locking plate receives the MJPEG images or pseudo-MJPEG images sent by the multi-channel encapsulation module through the UVC protocol and processes them to obtain corresponding video streams of different video formats; the rear locking plate transmits the obtained video streams of different video formats to different display terminals.

[0015] According to the single-channel multi-stream video transmission method based on MJPEG proposed in this invention, multiple encoders of different video formats are installed in the camera processing module to encode the received raw images to obtain images of corresponding different video formats. A multi-channel encapsulation module in the camera processing module receives images of different video formats and encapsulates them into MJPEG or pseudo-MJPEG images. The rear locking board receives the MJPEG or pseudo-MJPEG images sent by the multi-channel encapsulation module via the UVC protocol and processes them to obtain video streams of corresponding different video formats. The rear locking board transmits the obtained video streams of different video formats to different display terminals. Therefore, even when the rear locking board only supports one MJPEG transmission, the camera processing module multiplexes multiple video streams for transmission, allowing the rear locking board to select appropriate format and resolution video streams for forwarding to the corresponding devices based on network conditions, thereby reducing hardware and development costs.

[0016] In addition, the single-channel multi-stream video stream transmission method based on MJPEG proposed above according to the present invention may also have the following additional technical features:

[0017] Optionally, the multiple encoders of different video formats include an MJPEG encoder, an H264 encoder, and an H265 encoder, so that the received raw image can be encoded by encoders of different video formats to obtain images of corresponding different video formats.

[0018] Optionally, encapsulating the images of different video formats in an MJPEG image or a pseudo-MJPEG image includes adding a private protocol to the extended tag segment in the tag code of the MJPEG image or the pseudo-MJPEG image, so as to encapsulate the images of different video formats in the extended tag segment. Attached Figure Description

[0019] Figure 1 This is a block diagram of a single-channel multi-stream video transmission device based on MJPEG according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating a single-channel multi-stream video transmission method based on MJPEG according to an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the operation flow of the camera processing module according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the operation process of the rear side panel according to an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the pseudo-MJPEG encapsulation operation process according to an embodiment of the present invention. Detailed Implementation

[0024] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0025] Currently, smart locks connect the camera module and the rear lock panel via the UVC protocol. Additionally, real-time video playback via the cloud is required, typically in H264 or H265 format, which features low bitrate and low bandwidth consumption. If video transcoding is performed via the rear lock panel, it must support H264 encoding and decoding capabilities at a rate of at least 20 frames per second. However, in reality, the rear lock panel's processing capacity is limited, only supporting MJPEG encoding. Based on this understanding and research, three solutions are proposed: 1. Upgrade the rear lock panel; 2. Place the video stream forwarding function in the camera processing module; 3. Encode the H264 video stream in the camera processing module and transmit it to the rear lock panel via UVC. The first solution has high hardware and development costs; the second solution requires deploying cloud functionality in the camera module, which also incurs high development costs and requires the camera module to have network capabilities; the third solution has the advantages of low development and hardware costs.

[0026] To address this, the present invention proposes a single-channel multi-stream video transmission device based on MJPEG. This device incorporates multiple encoders and multi-encapsulation modules for different video formats on the camera processing module, with each encoder connected to the multi-encapsulation module. A rear locking plate is connected to the camera processing module to communicate with the multi-encapsulation module via the UVC protocol. Thus, even when the rear locking plate only supports one MJPEG transmission stream, the multi-encapsulation module multiplexes multiple video streams for transmission within the camera processing module. This allows the rear locking plate to select the appropriate format and resolution video stream for forwarding to the corresponding device based on network conditions, thereby reducing hardware and development costs.

[0027] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0028] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0029] refer to Figure 1 As shown, the single-channel multi-channel video stream transmission device based on MJPEG proposed in this embodiment of the invention includes a camera processing module 10 and a rear locking plate 20.

[0030] The camera processing module 10 includes multiple encoders 101 with different video formats and a multi-channel encapsulation module 102. The multiple encoders 101 with different video formats are connected to the multi-channel encapsulation module 102 respectively. The rear locking plate 20 is connected to the camera processing module 10 so as to communicate with the multi-channel encapsulation module 101 in the camera processing module 10 through the UVC protocol.

[0031] As one example, the encoders 101 for multiple different video formats include an MJPEG encoder, an H264 encoder, and an H265 encoder.

[0032] In other words, the camera sends the acquired NV21 raw image to the camera processing module 10, so that the MJPEG encoder in the camera processing module 10 encodes the received raw image to obtain an MJPEG image, the H264 encoder encodes the received raw image to obtain an H264 image, and the H265 encoder encodes the received raw image to obtain an H265 image. Then, the multiplexing module 102 receives the MJPEG image, H264 image, and H265 image, and encapsulates the H264 image in a pseudo-MJPEG image and the H265 image in a pseudo-MJPEG image, so that they can be transmitted to the rear locking plate 20 for processing via the UVC protocol.

[0033] It should be noted that the captured raw images are encoded in MJPEG, H264, and H265 formats respectively in the camera processing module. As an example, when encoding H264, if multiple video streams are supported, H264 streams with different resolutions and bitrates can be encoded, such as one stream being 1080p and another being CIF.

[0034] As an example, a private protocol is added to the extended tag segment in the tag code of an MJPEG image or pseudo-MJPEG image to encapsulate images of different video formats in the extended tag segment.

[0035] It should be noted that MJPEG is a dynamic image compression standard based on the JPEG compression standard for still images. The JPEG marker code consists of 2 bytes, with the first byte being a fixed value of 0xFF. The main marker codes include SOI (0xFFD8), APP0 (0xFFE0), APPn (0xFFE1-0xFFEF), DQT (0xFFDB), SOF0 (0xFFC0), DHT (0xFFC4), DRI (0xFFDD), SOS (0xFFDA), EOI (0xFFD9), etc. Among them, SOI indicates the start of the image, and EOI indicates the end of the image. APPn is an extended marker segment, in which a private protocol is added to encapsulate the H.264 bitstream data. This ensures that the MJPEG image is complete and valid. In addition to H.264 data, the private protocol can add information such as image resolution and bitrate.

[0036] Additionally, as an example, when encapsulating H264 streams, multiplexing methods can encapsulate each H264 stream within an encoded MJPEG image; or encapsulate multiple H264 streams within an encoded MJPEG image, where the distinction between each stream can be achieved by adding a segmentation identifier and a stream marker in a proprietary protocol; or encapsulate one or more H264 streams within a pseudo-MJPEG image.

[0037] The pseudo-MJPEG image contains only the following markers: SOI, APP0, APPn, and EOI, to ensure the validity of the MJPEG; however, it does not contain valid MJPEG image data.

[0038] It should be noted that, considering the complexity and convenience of parsing the bitstream on the rear lock plate 20, it is recommended to encapsulate each H264 bitstream as a pseudo-MJPEG image. During transmission, if the H264 bitstream is encapsulated in an MJPEG image, multiplexing of multiple bitstreams is required when transmitting the MJPEG image. During transmission, if the H264 bitstream is encapsulated in a pseudo-MJPEG image, the encoded H264 bitstream can be transmitted in real time without waiting for the MJPEG encoded frame.

[0039] As a specific implementation, during pseudo-MJPEG encapsulation, an image buffer is first allocated, with a size equal to H264 data plus MJPEG-related information. Then, relevant information is written to the buffer sequentially according to the MJPEG format: the SOI identifier FFD8 is written; the APP0 identifier and data are written, with the following valid format:

[0040] 0xFF,0xE0,0x00,0x10,0x4A,0x46,0x49,0x46,0x00,0x01,0x01,0x00,0x00,0x01,0x00,0x01,0x00,0x00. Where 0x4A,0x46,0x49,0x46,0x00 are the ASCII codes for JFIF, referencing the MJPEG protocol; write the APPn identifier, such as FFE7; write H.264 data and information; the private protocol can be shown in the table below:

[0041]

[0042]

[0043] Write the EOI identifier FFD9; End.

[0044] In summary, the encoded H264, H265, and other data are encapsulated in the MJPEG image format, supporting the multiplexing of one or more video streams within the same MJPEG frame. Alternatively, it can be encapsulated as pseudo-MJPEG frame data, which is a frame conforming to the MJPEG syntax format but does not actually carry MJPEG data, instead carrying H264 or other frame data. Pseudo-MJPEG frames are beneficial for real-time transmission, as they do not need to wait for the MJPEG frame data to complete before being transmitted to the back lock board 20 for processing via the UVC protocol.

[0045] As one embodiment, the rear locking plate 20 is used to receive MJPEG images and pseudo-MJPEG images in order to obtain the corresponding MJPEG video stream, H264 video stream and H265 video stream.

[0046] In other words, the rear locking plate 20 receives MJPEG images and pseudo-MJPEG images through the UVC transmission protocol and extracts the corresponding video streams for transmission.

[0047] As one embodiment, the MJPEG-based single-channel multi-channel video stream transmission device further includes a cloud 30, a first display device 40, and a second display device 50. The cloud 30, the first display device 40, and the second display device 50 are respectively connected to the rear locking plate 20 so that the rear locking plate 20 transmits the received H264 video stream to the cloud 30, transmits the received MJPEG video stream to the first display device 40, and transmits the received H265 video stream to the second display device 50.

[0048] It should be noted that UVC stands for USB Video Class, which is a protocol standard defined for USB video capture devices. The transmission channel provided in this invention complies with the UVC standard.

[0049] In addition, such as Figure 1As shown, the first display device 40 can be a display screen, and the second display device 50 can be other display devices, that is, display devices other than the display screen.

[0050] In summary, the single-channel multi-stream video stream transmission device based on MJPEG proposed in this invention, by setting multiple encoders and multi-encapsulation modules of different video formats on the camera processing module, and connecting the encoders of different video formats to the multi-encapsulation modules respectively; the rear locking plate is connected to the camera processing module to communicate with the multi-encapsulation modules in the camera processing module via the UVC protocol; thus, when the rear locking plate only supports one MJPEG transmission, the camera processing module multiplexes multiple video streams for transmission, allowing the rear locking plate to select the appropriate format and resolution video stream to forward to the corresponding device according to the network conditions, thereby reducing hardware and development costs.

[0051] In addition, such as Figure 2 As shown in the figure, this invention also proposes a single-channel multi-stream video stream transmission method based on MJPEG, including the following steps:

[0052] S101, multiple encoders of different video formats are set in the camera processing module to encode the received raw images to obtain images of corresponding different video formats.

[0053] S102, the multi-channel encapsulation module of the camera processing module receives images of different video formats and encapsulates the images of different video formats into MJPEG images or pseudo-MJPEG images.

[0054] In other words, such as Figure 3 As shown, multiple encoders for different video formats, including MJPEG encoders, H264 encoders, and H265 encoders, are used to encode the received raw images using encoders of different video formats to obtain images of the corresponding different video formats.

[0055] S103, the rear locking board receives MJPEG or pseudo-MJPEG images sent by the multi-channel encapsulation module via the UVC protocol, and processes them to obtain video streams of different corresponding video formats.

[0056] In other words, such as Figure 4 As shown, the rear locking plate 20 transmits the received H264 video stream to the cloud 30, the received MJPEG video stream to the first display device 40, and the received H265 video stream to the second display device 50.

[0057] S104, the rear locking plate transmits the video streams of different video formats to different display terminals accordingly.

[0058] As an example, a private protocol is added to the extended tag segment in the tag code of an MJPEG image or pseudo-MJPEG image to encapsulate images of different video formats in the extended tag segment.

[0059] As a specific example, such as Figure 5 As shown, identifiers are written sequentially, and a private protocol is added to the extended tag segment to encapsulate images of different video formats in the extended tag segment.

[0060] It should be noted that the above regarding Figure 1 The description of the MJPEG-based single-channel multi-channel video stream transmission device also applies to this MJPEG-based single-channel multi-channel video stream transmission method, and will not be repeated here.

[0061] According to the single-channel multi-stream video transmission method based on MJPEG proposed in this invention, multiple encoders of different video formats are installed in the camera processing module to encode the received raw images to obtain images of corresponding different video formats. A multi-channel encapsulation module in the camera processing module receives images of different video formats and encapsulates them into MJPEG or pseudo-MJPEG images. The rear locking board receives the MJPEG or pseudo-MJPEG images sent by the multi-channel encapsulation module via the UVC protocol and processes them to obtain video streams of corresponding different video formats. The rear locking board transmits the obtained video streams of different video formats to different display terminals. Therefore, even when the rear locking board only supports one MJPEG transmission, the camera processing module multiplexes multiple video streams for transmission, allowing the rear locking board to select appropriate format and resolution video streams for forwarding to the corresponding devices based on network conditions, thereby reducing hardware and development costs.

[0062] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0063] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0064] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0065] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0066] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0067] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A single channel multi-path video stream transmission device based on MJPEG, characterized in that, The application relates to a camera processing module, a rear lock plate, a cloud, a first display device and a second display device. The camera processing module comprises a plurality of different video format encoders and a multiplexing module, wherein the plurality of different video format encoders are connected to the multiplexing module. The rear lock plate is connected to the camera processing module to communicate with the multiplexing module in the camera processing module through a UVC protocol. The plurality of different video format encoders comprise an MJPEG encoder, an H264 encoder and an H265 encoder. The MJPEG encoder is used to encode a received original image to obtain an MJPEG image, the H264 encoder is used to encode the received original image to obtain an H264 image, and the H265 encoder is used to encode the received original image to obtain an H265 image. The multiplexing module is used to receive the MJPEG image, the H264 image and the H265 image, and encapsulate the H264 image in a pseudo-MJPEG image and the H265 image in a pseudo-MJPEG image. An extension tag section in a tag code of the MJPEG image or the pseudo-MJPEG image is added with a private protocol to encapsulate the different video format images in the extension tag section. In the pseudo-MJPEG image, a marker SOI, APP0, APPn and EOI are included to ensure the legality of the MJPEG, but no valid MJPEG image data is included.

2. The single channel multi-path video streaming apparatus based on MJPEG according to claim 1, wherein, The rear lock plate is used to receive the MJPEG image and the pseudo-MJPEG image to obtain corresponding MJPEG video stream, H264 video stream and H265 video stream.

3. The single channel multi-path video streaming apparatus based on MJPEG according to claim 2, wherein, The cloud, the first display device and the second display device are connected to the rear lock plate, so that the rear lock plate transmits the received H264 video stream to the cloud, transmits the received MJPEG video stream to the first display device and transmits the received H265 video stream to the second display device.

4. A single channel multi-path video streaming method based on MJPEG, characterized in that, The application further comprises the following steps: The plurality of different video format encoders arranged in the camera processing module encode a received original image to obtain corresponding different video format images. The multiplexing module arranged in the camera processing module receives the different video format images and encapsulates the different video format images in an MJPEG image or a pseudo-MJPEG image. The rear lock plate receives the MJPEG image or the pseudo-MJPEG image sent by the multiplexing module through a UVC protocol and processes the MJPEG image or the pseudo-MJPEG image to obtain corresponding different video format video streams. The rear lock plate transmits the obtained different video format video streams to different display terminals. The plurality of different video format encoders comprise an MJPEG encoder, an H264 encoder and an H265 encoder, so that the received original image is processed by the different video format encoders to obtain corresponding different video format images. Corresponding encapsulation of the images of different video formats in an MJPEG image or a pseudo-MJPEG image, comprising: Adding a private protocol in an extension tag section in the tag code of the MJPEG image or the pseudo-MJPEG image, so as to encapsulate the images of different video formats in the extension tag section; In the pseudo-MJPEG image, the tag symbols SOI, APP0, APPn, and EOI are included to ensure the legality of the MJPEG; But no valid MJPEG image data is included.

Images