Video image data transmission method and device based on MQTT protocol

Abstract

This application provides a video image data transmission method and apparatus based on the MQTT protocol. The method includes: responding to a connection request sent by a user terminal, sending an instruction to an associated video acquisition terminal to enable the video acquisition terminal to acquire real-time video image data; performing serialization processing on the video image data to obtain compressed video image data; receiving the serialized compressed video image data uploaded by the video acquisition terminal via the MQTT protocol, and performing secondary processing on the serialized compressed video image data using an image transmission processing unit cluster; and sending the secondary-processed compressed video image data to the user terminal via the MQTT protocol, so that the user terminal can stitch the secondary-processed compressed video image data together and play the stitched compressed video image data. This application reduces the computing power and bandwidth requirements of video acquisition terminals for video image transmission technology, reduces the transmission cost of video images, and improves the efficiency of image transmission.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for transmitting video image data based on the MQTT protocol. Background Technology

[0002] Wireless video transmission technology uses wireless cameras to capture video images and transmits them to a client via a network. Users can view video images of a target area in real time through the client. Wireless video transmission technology is widely used in fields such as drones and smart homes.

[0003] In existing technologies, wireless video transmission places high demands on the computing power of video acquisition devices, the processing capabilities of development boards, and network bandwidth. However, in some current scenarios, video acquisition devices lack sufficient computing power, and development boards have relatively low processing capabilities. This prevents existing wireless video transmission technologies from transmitting video from these devices. Instead, video transmission can only be achieved by upgrading the development boards and computing power of the video acquisition devices. This not only increases the cost of video and image transmission but also reduces the application scope of wireless video transmission technology. Therefore, existing technologies cannot achieve wireless video transmission in low-cost, low-computing-power, and low-bandwidth scenarios. Summary of the Invention

[0004] In view of this, embodiments of this application provide a video image data transmission method and apparatus based on the MQTT protocol to solve the problems of high transmission cost of video images and limited application scope of wireless video transmission technology in the prior art, which makes it impossible to achieve wireless video transmission in low-cost, low-computing-power, and low-bandwidth scenarios.

[0005] A first aspect of this application provides a video image data transmission method based on the MQTT protocol, comprising: responding to a connection request sent by a user terminal, sending an instruction to an associated video acquisition terminal to enable the video acquisition terminal to acquire real-time video image data; performing serialization processing on the video image data to obtain serialized compressed video image data; receiving the serialized compressed video image data uploaded by the video acquisition terminal via the MQTT protocol, and performing secondary processing on the serialized compressed video image data using an image transmission processing unit cluster; and sending the secondary processed compressed video image data to the user terminal via the MQTT protocol so that the user terminal can stitch the secondary processed compressed video image data together and play the stitched compressed video image data.

[0006] A second aspect of this application provides a video image data transmission device based on the MQTT protocol, comprising: an acquisition module configured to send an instruction to an associated video acquisition terminal in response to a connection request sent by a user terminal, so that the video acquisition terminal acquires real-time video image data; a serialization module configured to perform serialization processing on the video image data to obtain serialized compressed video image data; a secondary processing module configured to receive the serialized compressed video image data uploaded by the video acquisition terminal via the MQTT protocol, and perform secondary processing on the serialized compressed video image data using an image transmission processing unit cluster; and a splicing module configured to send the secondary processed compressed video image data to the user terminal via the MQTT protocol, so that the user terminal splices the secondary processed compressed video image data and plays the spliced ​​compressed video image data.

[0007] According to a third aspect of the present application, an electronic device is provided, including a processor and a memory, the memory being used to store a computer program that, when executed by the processor, implements the method of any of the above embodiments.

[0008] According to a fourth aspect of the present application, a computer-readable storage medium is provided, which stores a computer program, characterized in that the computer program, when executed by a processor, implements the method of any of the above embodiments.

[0009] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0010] In response to a connection request from a user terminal, an instruction is sent to the associated video acquisition terminal to enable the video acquisition terminal to acquire real-time video image data. The video image data is then serialized to obtain compressed video image data. The serialized compressed video image data uploaded by the video acquisition terminal via the MQTT protocol is received, and the serialized compressed video image data is further processed using an image transmission processing unit cluster. The secondary-processed compressed video image data is then sent to the user terminal via the MQTT protocol, allowing the user terminal to stitch the secondary-processed compressed video image data together and play the stitched compressed video image data. This application, by serializing video image data and using the MQTT protocol for video image data transmission, reduces the computational power and bandwidth requirements of the video acquisition terminal, lowers the transmission cost of video images, and is suitable for real-time video image transmission in low-bandwidth, low-computing-power, and low-cost environments. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of a video image data transmission system involved in a real-world scenario according to an embodiment of this application;

[0013] Figure 2 This is a flowchart illustrating the video image data transmission method based on the MQTT protocol provided in an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of the structure of a video image data transmission device based on the MQTT protocol provided in an embodiment of this application;

[0015] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0017] As described in the background section, existing wireless video transmission technologies place high demands on the computing power and network bandwidth of the acquisition terminal, as well as the processing power of the development board. Therefore, they cannot achieve image transmission for video acquisition devices in low-cost, low-computing-power, and low-bandwidth scenarios. The high demands on the computing power of the video acquisition device, the processing power of the development board, and the network bandwidth of the existing wireless video transmission technology increase the cost of video and image transmission and significantly reduce the scope of application scenarios for wireless video transmission technology. Therefore, how to reduce the dependence of wireless video transmission technology on the processing power of the development board and network bandwidth while using acquisition terminals with low-cost, low-computing-power development boards is one of the important problems that needs to be solved in the field of wireless video transmission technology.

[0018] In view of the problems existing in the prior art, this application provides a video image data transmission method based on the MQTT protocol. This invention utilizes the Protobuf method to serialize video images and transmits the serialized compressed video image data via the MQTT protocol. The image transmission processing unit cluster in the video streaming server performs secondary processing on the serialized compressed video image data. The user end then splices and plays the secondary-processed compressed video image data, thereby reducing network bandwidth and the cost of the video acquisition terminal. The image transmission processing unit processes the video images, and based on the load pressure of the image transmission processing unit cluster, nodes are created or destroyed using a K8s cluster approach. This dynamic management of K8s cluster nodes improves the image processing efficiency of the image transmission processing unit cluster. Destroying a node releases server resources to a resource pool, achieving rational resource utilization.

[0019] The system structure involved in the practical application of the technical solution of this application will be described in detail below with reference to the accompanying drawings. Figure 1 This is a schematic diagram of the structure of a video image data transmission system involved in a real-world scenario according to an embodiment of this application. For example... Figure 1 As shown, the video image data transmission system may specifically include:

[0020] The video image data transmission system consists of three parts: a video acquisition terminal, a video streaming server, and a user terminal. The video acquisition terminal is used to acquire video images of the target. The video acquisition terminal is equipped with a camera and can transmit the acquired video images to the video streaming server via a network. In practical applications, the video acquisition terminal in this embodiment can be a low-cost, low-computing-power device terminal, such as a robot vacuum cleaner, a cat litter box, or other acquisition devices with video shooting capabilities.

[0021] A video streaming server is used for the transmission and processing of image data. It uses an Image Transmission Processing Unit Cluster (EMQ Cluster) to perform secondary processing on the compressed video image data sent by the video capture terminal, and then sends the secondary-processed compressed video image data to the user terminal via the MQTT protocol. The video streaming server may include a user management module, a first Image Transmission Processing Unit Cluster (referred to as the first EMQ Cluster), and a second Image Transmission Processing Unit Cluster (referred to as the second EMQ Cluster). The user management module is used for registering video capture terminals, establishing the correspondence between video capture terminals and user terminals, and verifying login requests from user terminals. The first Image Transmission Processing Unit Cluster is used for transmitting and processing structured data, including the IP address, remaining battery power, and real-time status information of the video capture terminal. The second Image Transmission Processing Unit Cluster is used for transmitting and processing compressed video image data, including operations such as face detection, scene detection, and image enhancement. Both the first and second Image Transmission Processing Unit Clusters consist of one or more Image Transmission Processing Units (EMQs). In practical applications, a video streaming server can also be called a streaming media server.

[0022] The client is used to receive compressed video image data after secondary processing sent by the video streaming server via the MQTT protocol. The client stitches the image data together and outputs it to the player for display. In other words, the client pulls the corresponding video stream file from the video streaming server and plays the pulled video stream file through the APP. In practical applications, the client can be a mobile terminal (such as a mobile phone, tablet, etc.) or a PC.

[0023] The video image data transmission method based on the MQTT protocol provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Figure 2 This is a flowchart illustrating the video image data transmission method based on the MQTT protocol provided in the embodiments of this application. Figure 2 The MQTT-based video image data transmission method can be executed by a video streaming server. For example... Figure 2 As shown, the video image data transmission method based on the MQTT protocol may specifically include:

[0025] S201, in response to the connection request sent by the user terminal, sends an instruction to the associated video acquisition terminal so that the video acquisition terminal can acquire real-time video image data;

[0026] S202, Perform serialization processing on the video image data to obtain serialized compressed video image data;

[0027] S203 receives the serialized compressed video image data uploaded by the video acquisition terminal via the MQTT protocol, and performs secondary processing on the serialized compressed video image data using the image transmission processing unit cluster.

[0028] S204: The secondary processed compressed video image data is sent to the user terminal via the MQTT protocol, so that the user terminal can stitch the secondary processed compressed video image data together and play the stitched compressed video image data.

[0029] Specifically, the video acquisition terminal in this application includes some low-cost, low-computing-power embedded board acquisition device terminals, and can realize real-time transmission of video images in low-bandwidth scenarios. Since this application does not have high requirements for the computing power and storage of the video acquisition terminal, low-cost, low-bandwidth acquisition devices can be used to acquire targets, such as robot vacuum cleaners, cat litter boxes and other acquisition devices with wireless camera functions.

[0030] Furthermore, Message Queuing Telemetry Transport (MQTT) is a lightweight IoT instant messaging protocol based on a publish / subscribe model. The MQTT protocol provides reliable network services for IoT devices in low-bandwidth and unstable network environments. It is suitable for low-cost, low-computing-power embedded modules to use the MQTT protocol for image transmission, reducing bandwidth network requirements and saving costs. Based on the MQTT protocol, video image transmission can be implemented in low-bandwidth environments and on low-computing-power development boards.

[0031] In some embodiments, before responding to a connection request sent by a user terminal, the method further includes: receiving a registration request sent by a video acquisition terminal using a user management module in a video streaming server, registering the video acquisition terminal based on the registration request, and establishing an association between the video acquisition terminal identifier and the user terminal identifier.

[0032] Specifically, in order to determine the connection relationship between the video capture terminal and the user terminal and to prevent user terminals that are not friends from connecting to the video capture terminal, this embodiment first registers the video capture terminal with the video streaming server (i.e., the streaming media server). After successfully registering with the video streaming server, the video capture terminal can send a heartbeat to the video streaming server every certain period of time. After receiving the heartbeat, the video streaming server sends a message to the video capture terminal to notify the video capture terminal that the video streaming server is currently online.

[0033] In practical applications, after the video acquisition terminal is registered through the user management module, the user management module can establish an association between the video acquisition terminal identifier and the user terminal identifier.

[0034] In some embodiments, after establishing the association between the video acquisition terminal identifier and the user terminal identifier, the method further includes: receiving a login request sent by the user terminal, verifying the login request using the user management module, and, after successful verification, establishing a connection between the user terminal and the video acquisition terminal based on the association between the video acquisition terminal identifier and the user terminal identifier.

[0035] Specifically, users can log in to their accounts through the user management module. By entering their username and password on the login interface provided by the APP, the APP generates a login request based on the username and password and sends the login request to the user management module via the network. The user management module verifies the user login request by querying the relationship chain information based on the stored relationship chain of usernames and passwords. After successful verification, based on the pre-established association between the video acquisition terminal identifier and the user terminal identifier, the user terminals and video acquisition terminals with friend relationships are connected to the image transmission and processing unit cluster.

[0036] It should be noted that the user client only establishes a connection with the image transmission and processing unit cluster when logging in. The video streaming server receives the connection request sent by the user client, uses the image transmission and processing unit cluster to receive the compressed video image data uploaded by the video acquisition terminal, and transmits the compressed video image data to the user client in real time for playback, so that the user can watch smooth video images.

[0037] In some embodiments, performing serialization processing on video image data to obtain serialized compressed video image data includes: using the Protobuf serialization protocol to serialize the video image data in order to compress the video image data and obtain serialized compressed video image data.

[0038] Specifically, before transmitting video image data using the MQTT protocol, the acquired video image data is first serialized using the Protobuf serialization protocol. After serialization, the size of the video image data is reduced, thereby reducing network I / O and lowering the network bandwidth requirements for image data transmission, thus enabling real-time transmission of video images with low bandwidth and low computing power.

[0039] Protobuf is a lightweight and efficient structured data storage format that can be used for structured data serialization or serialization. It is suitable for data storage or RPC data exchange and can be used in instant messaging, data storage, and other fields as a language-independent, platform-independent, and scalable serialized structured data format.

[0040] This application serializes video image data using the Protobuf serialization protocol. Protobuf employs Varint and ZigZa compression algorithms, making the compressed binary video image data very compact. Using Protobuf for video image data serialization significantly reduces data volume and network I / O, thereby reducing network transmission time. Therefore, this application's embodiments are suitable for video image data transmission in low-bandwidth environments. In practical applications, besides using the Protobuf serialization protocol to compress video image data, other serialization methods can also be used to serialize images, such as Thrift and Avro protocols.

[0041] In some embodiments, the image transmission processing unit cluster includes a first image transmission processing unit cluster and a second image transmission processing unit cluster. The method further includes: using the first image transmission processing unit cluster to receive structured data uploaded by the video acquisition terminal via the MQTT protocol, and sending the structured data to the user terminal; wherein the structured data includes the IP address, remaining battery power, and real-time status information of the video acquisition terminal.

[0042] Specifically, after the video capture terminal serializes the video image data using Protobuf, the embedded development board of the video capture terminal uploads the processed data to the video stream server. The data uploaded to the video stream server includes two aspects: one is the structured data generated by the video capture terminal, such as the IP address, remaining battery power, and real-time status information of the video capture terminal; the other is the compressed video image data after Protobuf serialization. In this embodiment, the video stream server is configured with two image transmission processing unit clusters, used to receive and process the aforementioned two aspects of data respectively. The first image transmission processing unit cluster (i.e., the first EMQ cluster) receives the structured data uploaded by the video capture terminal via the MQTT protocol and establishes an association between the user terminal and the video capture terminal through the structured data. The second image transmission processing unit cluster (i.e., the second EMQ cluster) receives the compressed video image data and performs secondary processing on it. Since the structured data and compressed video image data are transmitted and processed separately using two EMQ clusters, the transmission of image data does not affect the transmission of structured data, thus improving the data processing efficiency of the EMQ clusters.

[0043] In some embodiments, the image transmission processing unit cluster is used to perform secondary processing on the serialized compressed video image data, including: using the second image transmission processing unit cluster to establish a streaming relationship between the video acquisition terminal and multiple user terminals based on the connection relationship between the user terminal and the video acquisition terminal; and / or, using the second image transmission processing unit cluster to perform face detection, scene detection and image enhancement operations on the serialized compressed video image data.

[0044] Specifically, after receiving compressed video image data uploaded by the video capture terminal via the MQTT protocol, the second EMQ cluster will perform secondary processing on the compressed video image data. This secondary processing includes two aspects: first, enabling multi-user terminal streaming, allowing multiple user terminals to simultaneously request streaming of video image data from the same video capture terminal; second, performing secondary processing on the compressed video image data, such as face detection, scene detection, and image enhancement. Through multiple EMQs (i.e., image transmission and processing units) in the second EMQ cluster, the compressed video image data is processed and then transmitted to the user terminal, thereby achieving secondary enhancement of the video images.

[0045] It should be noted that the first and second EMQ clusters in this application embodiment contain multiple EMQ units. Each EMQ unit in the EMQ cluster can process video image data from different video acquisition terminals. By using a cluster unit approach to transmit and process video images, the efficiency of video image transmission and processing can be improved. Data transmission between the EMQ cluster, the user terminal, and the video acquisition terminal all use the MQTT communication protocol. Therefore, the video image transmission in this application embodiment has the advantages of high throughput, low latency, link multiplexing, increased bandwidth to improve transmission capacity, and improved data transmission efficiency.

[0046] In practical applications, before using the second EMQ cluster to transmit and process video image data, the second EMQ cluster can be initialized according to the number of connected video acquisition terminals. That is, the number of EMQ units in the second EMQ cluster is reconfigured according to the number of connected video acquisition terminals. The EMQ units provide video image transmission services to the user end.

[0047] In some embodiments, the method further includes: using a load monitoring program set in the image transmission processing unit cluster to monitor the network load and server load of the image transmission processing unit, and dynamically creating and releasing image transmission processing units in the image transmission processing unit cluster based on the monitoring results.

[0048] Specifically, in order to achieve dynamic expansion and destruction of image transmission processing units, thereby dynamically adjusting server resources and making reasonable use of server resources, this application uses Kubernetes (K8s) to manage the unit nodes in the EMQ cluster. K8s is an open source application used to manage containerized applications on multiple hosts in a cloud platform. K8s can be considered a container orchestrator. K8s creates or destroys image transmission processing units through container management, thereby managing the EMQ cluster.

[0049] Furthermore, this application utilizes Kubernetes to monitor the network load (network I / O) and server load (EMQ load) of the image transmission processing unit. Based on the monitoring results, it dynamically expands and reclaims the image transmission processing unit, ensuring that the unit can rationally utilize server resources, making the best use of server resources, and achieving reasonable allocation of server resources. The following detailed description of the load monitoring and dynamic management process of the image transmission processing unit, using specific embodiments, includes the following:

[0050] By configuring a load monitoring program within the EMQ cluster, the program monitors the network I / O and EMQ load of the image transmission processing units. It dynamically manages these units based on real-time load changes. For example, when network bandwidth utilization exceeds 90%, it increases network bandwidth. Simultaneously, it monitors the server load of each EMQ unit. When both network and server loads fall below 50%, it appropriately releases EMQ units. A minimum number of EMQs is set (e.g., a minimum of 10 EMQs), and some idle EMQ units are reserved. When expanding EMQ units is needed, idle EMQ units are directly used for image data transmission processing. In practical applications, expanding an EMQ unit can be understood as creating a new EMQ unit or using an idle EMQ unit, while reclaiming an EMQ unit can be understood as releasing the computing and storage resources occupied by the EMQ unit.

[0051] It should be noted that, in addition to using Kubernetes to manage image transmission processing units, other containerization management tools can also be used, such as Ranger to manage containers. A container contains one or more image transmission processing units, and one or more image transmission processing units can be created or released at once through containers.

[0052] According to the technical solution provided in the embodiments of this application, after serializing video images using Protobuf, the serialized compressed video image data is uploaded to a video streaming server using the MQTT communication protocol. The EMQ cluster in the video streaming server processes the compressed video image data. Since the video acquisition terminal transmits compressed video image data, the requirements for the computing power and network bandwidth of the development board are reduced, enabling real-time transmission of video images in low-bandwidth, low-computing-power client environments. The video streaming server uses an EMQ cluster to perform secondary processing on the compressed video image data, ensuring the quality of the video image after secondary processing. Furthermore, by monitoring the network load and server load of the image transmission processing unit, the image transmission processing unit is dynamically increased or decreased through container management, thereby achieving reasonable allocation of server resources. This application reduces the computing power and bandwidth requirements of video acquisition terminals for video transmission technology, reduces the transmission cost of video images, expands the application scenarios of video transmission technology, and facilitates the promotion and application of video transmission technology.

[0053] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0054] Figure 3 This is a schematic diagram of the structure of a video image data transmission device based on the MQTT protocol provided in an embodiment of this application. Figure 3 As shown, the video image data transmission device based on the MQTT protocol includes:

[0055] The acquisition module 301 is configured to send an instruction to the associated video acquisition terminal in response to a connection request sent by the user terminal, so that the video acquisition terminal can acquire real-time video image data.

[0056] The serialization module 302 is configured to perform serialization processing on video image data to obtain serialized compressed video image data.

[0057] The secondary processing module 303 is configured to receive serialized compressed video image data uploaded by the video acquisition terminal via the MQTT protocol, and to perform secondary processing on the serialized compressed video image data using the image transmission processing unit cluster.

[0058] The splicing module 304 is configured to send the secondary processed compressed video image data to the user terminal via the MQTT protocol, so that the user terminal can splice the secondary processed compressed video image data and play the spliced ​​compressed video image data.

[0059] In some embodiments, Figure 3Before responding to the connection request sent by the user terminal, the user management module 305 uses the user management module in the video stream server to receive the registration request sent by the video acquisition terminal, registers the video acquisition terminal based on the registration request, and establishes the association between the video acquisition terminal identifier and the user terminal identifier.

[0060] In some embodiments, Figure 3 After establishing the association between the video acquisition terminal identifier and the user terminal identifier, the user management module 305 receives the login request sent by the user terminal, verifies the login request using the user management module, and establishes the connection between the user terminal and the video acquisition terminal based on the association between the video acquisition terminal identifier and the user terminal identifier after successful verification.

[0061] In some embodiments, Figure 3 The serialization module 302 uses the Protobuf serialization protocol to serialize the video image data in order to compress the video image data and obtain compressed video image data after serialization.

[0062] In some embodiments, the image transmission processing unit cluster includes a first image transmission processing unit cluster and a second image transmission processing unit cluster. Figure 3 The structured data transmission module 306 uses the first image transmission processing unit cluster to receive structured data uploaded by the video acquisition terminal via the MQTT protocol, and sends the structured data to the user terminal; wherein, the structured data includes the IP address, remaining battery power and real-time status information of the video acquisition terminal.

[0063] In some embodiments, Figure 3 The secondary processing module 303 utilizes the second image transmission processing unit cluster to establish a streaming relationship between the video acquisition terminal and multiple user terminals based on the connection relationship between the user terminal and the video acquisition terminal; and / or, utilizes the second image transmission processing unit cluster to perform face detection, scene detection and image enhancement operations on the serialized compressed video image data.

[0064] In some embodiments, Figure 3 The unit management module 307 uses the load monitoring program set in the image transmission processing unit cluster to monitor the network load and server load of the image transmission processing unit, and dynamically creates and releases the image transmission processing units in the image transmission processing unit cluster based on the monitoring results.

[0065] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0066] Figure 4 This is a schematic diagram of the structure of the electronic device 4 provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 in this embodiment includes a processor 401 and a memory 402. The memory 402 stores a computer program 403 that can run on the processor 401. When the processor 401 executes the computer program 403, it implements the steps in the various method embodiments described above. Alternatively, when the processor 401 executes the computer program 403, it implements the functions of each module / unit in the various device embodiments described above.

[0067] For example, computer program 403 may be divided into one or more modules / units, which are stored in memory 402 and executed by processor 401 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 403 in electronic device 4.

[0068] Electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 4 may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0069] Processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0070] The memory 402 can be an internal storage unit of the electronic device 4, such as a hard disk or RAM. The memory 402 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 402 can include both internal and external storage units of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device. The memory 402 can also be used to temporarily store data that has been output or will be output.

[0071] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0072] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments claimed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0074] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0078] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A video image data transmission method based on the MQTT protocol, characterized in that, The method comprises: in response to a connection request sent by a user terminal, sending an instruction to an associated video capture terminal to make the video capture terminal obtain real-time video image data; performing serialization processing operation on the video image data to obtain serialized and compressed video image data; receiving the serialized and compressed video image data uploaded by the video capture terminal through MQTT protocol, and performing secondary processing on the serialized and compressed video image data using an image transmission processing unit cluster, the secondary processing including at least one of image enhancement, face detection or scene detection; sending the secondary-processed and compressed video image data to the user terminal through MQTT protocol to make the user terminal splice the secondary-processed and compressed video image data and play the spliced and compressed video image data; monitoring the network load and server load of the image transmission processing unit using a load monitoring program set in the image transmission processing unit cluster, and dynamically creating and releasing the image transmission processing unit in the image transmission processing unit cluster according to the monitoring result.

2. The method of claim 1, wherein, Before the response to the connection request sent by the user terminal, the method further comprises: receiving a registration request sent by the video capture terminal using a user management module in the video stream server, registering the video capture terminal based on the registration request, and establishing an association between the video capture terminal identifier and the user terminal identifier.

3. The method of claim 2, wherein, After the association between the video capture terminal identifier and the user terminal identifier is established, the method further comprises: receiving a login request sent by the user terminal, verifying the login request using the user management module, and after verification, establishing a connection relationship between the user terminal and the video capture terminal based on the association between the video capture terminal identifier and the user terminal identifier.

4. The method of claim 1, wherein, The serialization processing operation on the video image data to obtain serialized and compressed video image data comprises: serializing the video image data using Protobuf serialization protocol to compress the video image data and obtain serialized and compressed video image data.

5. The method of claim 1, wherein, The image transmission processing unit cluster comprises a first image transmission processing unit cluster and a second image transmission processing unit cluster, and the method further comprises: receiving structured data uploaded by the video capture terminal through MQTT protocol using the first image transmission processing unit cluster, and sending the structured data to the user terminal; wherein the structured data contains the IP address, remaining power and real-time state information of the video capture terminal.

6. The method of claim 5, wherein, The secondary processing on the serialized and compressed video image data using the image transmission processing unit cluster comprises: establishing a pull stream relationship between the video capture terminal and multiple user terminals based on the connection relationship between the user terminal and the video capture terminal using the second image transmission processing unit cluster; And / or, the second image transmission processing unit cluster is utilized to perform face detection, scene detection and image enhancement on the serialized and compressed video image data.

7. A video image data transmission device based on an MQTT protocol, characterized by, The method comprises the following steps: An acquisition module is configured to, in response to a connection request sent by a user terminal, send an instruction to an associated video capture terminal to cause the video capture terminal to acquire real-time video image data; A serialization module is configured to perform a serialization processing operation on the video image data to obtain serialized and compressed video image data; A secondary processing module is configured to receive the serialized and compressed video image data uploaded by the video capture terminal through an MQTT protocol, and utilize an image transmission processing unit cluster to perform secondary processing on the serialized and compressed video image data, the secondary processing including at least one of image enhancement, face detection or scene detection; A splicing module is configured to send the secondary-processed and compressed video image data to the user terminal through the MQTT protocol to cause the user terminal to splice the secondary-processed and compressed video image data and play the spliced and compressed video image data; A unit management module is configured to utilize a load monitoring program arranged in the image transmission processing unit cluster to monitor network load and server load of the image transmission processing unit, and dynamically create and release the image transmission processing unit in the image transmission processing unit cluster according to the monitoring result. 8.An electronic device comprising a processor and a memory, the memory being configured to store a computer program, the computer program being executed by the processor to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images