A multifunctional real-time imaging platform

Abstract

The application discloses a kind of multifunctional real-time imaging platform, support camera holder control, camera image parameter configuration, camera code stream parameter configuration, camera device parameter configuration, camera remote upgrade and so on function.Customer end is based on the way of HTTP+FLV+MSE to realize real-time pull stream, realization is simple, delay time can be controlled in 1-2s.Server uses open source framework ZLMediaKit to carry out FLV push stream, and performance is superior, realization is simple.Server is realized high concurrency model by thread pool, non-blocking socket, based on horizontal trigger's epoll and Reactor event processing model, can be better applied to high concurrency scene.The platform has good compatibility, can be easily transplanted to any camera that provides relevant function of SDK.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of imaging, video transmission, video monitoring, etc., and particularly relates to a multifunctional real-time imaging platform. BACKGROUND

[0002] Traditional video monitoring systems are relatively single in function and lack flexible control over cameras. This limitation makes it impossible for users to adjust video in real time according to actual needs, seriously affecting the effectiveness and flexibility of monitoring. In addition, most monitoring systems adopt the form of host computers, which have the disadvantages of poor cross-platform compatibility and inconvenient remote access.

[0003] In real-time video streaming technology, the traditional technical solution has a high delay problem, which is difficult to meet the needs of low-delay applications. The solutions based on RTMP (Real-Time Messaging Protocol) and HLS (HTTP Live Streaming) are common transmission technologies in the early stage. Although RTMP is relatively low in delay, it is gradually abandoned by modern browsers due to its dependence on Flash plug-ins. While the design of HLS is to provide stability and compatibility, but its fragmentation mechanism leads to a delay of usually 5-10 seconds or even higher, which cannot realize low-delay real-time video transmission.

[0004] In addition, the early video streaming solutions also have problems in browser compatibility. Many solutions rely on plug-ins or specific browser features, such as RTMP relying on Flash plug-ins. Flash plug-ins not only have security risks, but are gradually eliminated by major browsers. This leads to poor compatibility of these solutions in modern browsers, affecting the user's viewing experience and the universality of the system.

[0005] In summary, the traditional video monitoring system has obvious shortcomings in function, platform compatibility, remote access, and browser compatibility. These problems limit the application and development of the monitoring system, and the existing technical solutions need to be improved and optimized to provide more flexible, efficient and secure monitoring solutions. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of traditional video monitoring systems and provide a multifunctional real-time imaging platform that supports high concurrency, cross-platform compatibility, easy portability, strong real-time performance, and remote camera setting and upgrading.

[0007] The application mainly solves the problems of the traditional video monitoring system through the following ways: 1. using a Web terminal as a client, supporting multi-platform compatibility, remote camera setting and upgrading; 2. solving the problems of large delay and poor compatibility existing in the traditional video stream transmission technology based on the FLV stream media format; and 3. better applying to a high concurrency scene through a high-performance concurrency model.

[0008] The application aims to realize the following technical scheme: a multifunctional real-time imaging platform comprising a server and a client; the server is built based on a Linux system, and a concurrency model thereof is realized through a thread pool, a non-blocking socket, an epoll based on horizontal triggering and a Reactor event processing model; and a design process of the server and the client comprises:

[0009] (1) creating a thread pool, wherein a working thread, a stream pushing thread and an upgrading thread are included, and a maximum capacity of the working thread is preset for the thread pool;

[0010] (2) creating a socket through a socket, setting it as a non-blocking mode by using fcntl, then binding it to an IP address and a specified port of the server, and calling bind and listen to start listening to a client connection request; calling epoll_create to create an epoll instance, using epoll_ctl to register the socket to the epoll instance, and listening to readable events on the socket so as to notify a main thread when a new client connection request is received;

[0011] (3) in the main thread, listening to readable events on the socket through epoll_wait, if readable events on the socket are listened to, calling accept to accept a new client connection request, and returning a new socket; configuring the new socket as a non-blocking mode; using epoll_ctl to register the new socket to the epoll instance, listening to readable events on the new socket; when readable events on the new socket are listened to, calling recv through a loop until a complete HTTP request is read, and adding the read complete HTTP request to a request queue;

[0012] (4) obtaining a task from the request queue, parsing an HTTP request, generating a corresponding HTTP response according to specific content of the parsed HTTP request, and calling send through a loop to return the complete HTTP response to the client.

[0013] Further, in the thread pool, the working thread is used to process a client request, in order to ensure that a stream pushing function and an upgrading function of the server are not affected by the working thread, a stream pushing thread and an upgrading thread are separately created.

[0014] Further, the thread pool is set to a higher thread priority for the push thread and the upgrade thread to ensure priority when scheduling.

[0015] Further, in step (4), if the HTTP request is a URL corresponding to a camera video stream preview request, the server assigns the task to the push thread, calls the API interface of the camera SDK push stream in the push thread, converts the RTMP video stream into an FLV video stream through the open-source ZLMediaKit framework, places the FLV video stream in the body part of the HTTP response, calls send in a loop, returns the complete HTTP response to the client, and maintains the long connection to continue the continuous push stream.

[0016] Further, after the client receives the FLV video stream from the server, a new media extension source is created through MSE, and the new media extension source is associated with an HTML5 video element; flv.js is used to process the FLV video stream, which parses the FLV file, encodes and converts it into an MP4 segment, loads the MP4 segment into the media extension source, and the media extension source pushes the MP4 segment to the associated HTML5 video element for playback.

[0017] Further, in step (4), if the HTTP request is a URL corresponding to a Web login request, the server assigns the task to the worker thread, parses the account and password in the request body of the HTTP request in the worker thread, uses SQLite as the database, accesses the database by taking a connection from the database connection pool, compares and verifies whether the parsed account and password are equal to the corresponding account and password in the database, takes the verification result as the body part of the HTTP response, calls send in a loop, returns the complete HTTP response to the client, and if the judgment is equal, the client enters the main page, and if the judgment is not equal, the client stays on the login page.

[0018] Further, in step (4), if the HTTP request is a URL corresponding to a camera image parameter configuration request, a camera device parameter configuration request, or a camera code stream parameter configuration request, the server assigns the task to the worker thread; if the HTTP request is a GET request, the relevant API interface of the camera SDK is called to obtain relevant information and return it to the client; if the HTTP request is a POST request, the relevant API interface of the camera SDK is called to set the success result and return it to the client.

[0019] Further, in step (4), if the HTTP request is a URL corresponding to a camera SDK upgrade preprocessing request, the server assigns the task to the upgrade thread, and then completes the upgrade function according to the following sub-steps:

[0020] a) In the upgrade thread, put the upgrade preparation identification field in the body part of the HTTP response, and call send in a loop to return the complete HTTP response to the client, so as to inform the client that the server has prepared to receive the upgrade package and maintain the long connection;

[0021] b) The client receives the response returned in step a), reads the data segment of the preset length of the upgrade package file each time, puts the read data segment into the request body of the HTTP request, adds the preset length field in the header field of the HTTP request, and sends to the server through the POST method;

[0022] c) The server obtains the task from the request queue and parses the HTTP request, if it is a camera upgrade package, sends the corresponding URL, distributes the task to the upgrade thread, puts the upgrade package file in the request body of the HTTP request into the cache area, and accumulates the size of the received upgrade package file, during which the server does not make a related HTTP response reply;

[0023] d) After the client sends all the contents of the upgrade package file, the number of sent bytes is filled into the request body of the HTTP request as a check package, and is sent to the server through the POST method, for checking whether the upgrade package file received by the server is complete;

[0024] e) The server obtains the task from the request queue and parses the HTTP request, if it is a camera check package, checks the corresponding URL, distributes the task to the upgrade thread, compares the number of sent bytes in the request body of the HTTP request with the size of the received bytes in the cache area of the server, if they are equal, the check is passed, the upgrade API interface in the camera SDK is called, and the upgrade mode is entered; if they are not equal, the check fails, the server puts the upgrade failure identification field in the body part of the HTTP response, and calls send in a loop to return the complete HTTP response to the client, after the client receives the HTTP response, the upgrade retry is started according to step b), until the upgrade is successful.

[0025] Further, the upgrade API interface in the camera SDK is called to enter the upgrade mode, specifically, the upgrade package file received by the cache area is written into the firmware gradually, after writing, the camera is restarted and the upgrade package file is loaded, after the camera is started, the server puts the upgrade completion identification field in the body part of the HTTP response, and calls send in a loop to return the complete HTTP response to the client, so as to inform the client that the server has completed the upgrade, and close the current socket.

[0026] The beneficial effects of the application are as follows:

[0027] 1. The client realizes real-time pull streaming in the HTTP+FLV+MSE mode, which is simple in implementation and can control the delay time within 1-2s.

[0028] 2. The multifunctional real-time imaging platform supports camera holder control, camera image parameter setting, camera code stream parameter setting, camera device parameter setting and camera remote upgrading.

[0029] 3. The server realizes FLV push streaming by using the open source framework ZLMediaKit, which is superior in performance and simple in implementation.

[0030] 4. The server uses SQLite as a database to store some state data of the client, which is light in weight.

[0031] 5. The platform has good compatibility and can be easily transplanted to any camera providing the SDK of related functions.

[0032] 6. The server realizes a high concurrency model by using a thread pool, a non-blocking socket, an epoll based on level triggering and a Reactor event processing model, and can be better applied to a high concurrency scene. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0034] Figure 1 The embodiment of the present application provides the overall design framework of the server of the multifunctional real-time imaging platform.

[0035] Figure 2 The embodiment of the present application provides the overall design framework of the client of the multifunctional real-time imaging platform. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0037] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0038] The embodiment provides a multifunctional real-time imaging platform, which comprises a server and a client, and the overall design framework of the server and the client is respectively as shown in Figure 1 and Figure 2 .

[0039] The server is built based on a Linux system, and a concurrent model thereof is realized through a thread pool, a non-blocking socket, an epoll based on horizontal triggering and a Reactor event processing model. Specifically, the design process of the server and the client comprises the following steps:

[0040] 1. A thread pool is created, and the maximum capacity of working threads of the thread pool is preset. The working threads will be used for processing client requests, and in order to ensure that the push stream function and the upgrade function of the server are not affected by other working threads, a push stream thread and an upgrade thread are separately created, and a higher thread priority is set to ensure the priority in scheduling. Usually, the push stream thread and the upgrade thread have the same priority, and are higher than the working thread priority.

[0041] 2. A socket is created through a socket, and is set to a non-blocking mode by using fcntl, and then is bound to the IP address and the specified port of the server, and bind and listen are called to start listening to the client connection request. Then an epoll instance is created by calling epoll_create, and the socket is registered to the epoll instance by using epoll_ctl, and a readable event (EPOLLIN) on the socket is listened to, so as to notify the main thread when a new client connection request is received.

[0042] 3. In the main thread, the readable event on the socket is listened to through epoll_wait, if the readable event on the socket is listened to, a new client connection request is accepted by calling accept, and a new socket is returned. The returned new socket is also configured to a non-blocking mode. The new socket is registered to the epoll instance by using epoll_ctl, and the readable event on the new socket is listened to. When the readable event on the new socket is listened to, the complete HTTP request is read through a loop call of recv, and the read complete HTTP request is added to a request queue.

[0043] 4. A task is obtained from the request queue, and the HTTP request is parsed, including extracting the request line, the header and the request body. According to the specific content of the parsed HTTP request, a proper HTTP response is generated, and the complete HTTP response is returned to the client through a loop call of send.

[0044] 1) If the HTTP request is the URL corresponding to the camera video stream preview request, the server distributes the task to the push stream thread, in which the API interface of the camera SDK is called to push the stream, the open source ZLMediaKit framework is used to convert the RTMP video stream into the FLV video stream, the FLV video stream is placed in the body part of the HTTP response, the send is called in a loop, and the complete HTTP response is returned to the client, while the long connection is maintained to continue the continuous push stream.

[0045] After the client receives the FLV video stream of the server, a new media extension source is created through the MSE (Media Source Extensions), and the new media extension source is associated with the HTML5 video element. Then, flv.js is used to process the FLV video stream, which parses the FLV file, encodes and converts it into an MP4 segment, loads the MP4 segment into the media extension source, and the media extension source pushes the MP4 segment to the associated HTML5 video element for playback.

[0046] 2) If the HTTP request is the URL corresponding to the Web login request, the server distributes the task to the worker thread, in which the account and password in the request body of the HTTP request are parsed out, considering the limited resources of the embedded device, the lightweight SQLite is used as the database. A connection is taken out from the database connection pool to access the database, the parsed account and password are compared with the corresponding account and password in the database to verify whether they are equal, and the verification result is taken as the body part of the HTTP response, the send is called in a loop, and the complete HTTP response is returned to the client. If it is equal, that is, the verification is passed, the client enters the main page, and if it is not equal, that is, the verification fails, the client stays in the login page.

[0047] 3) If the HTTP request is the URL corresponding to the camera image parameter configuration request, the camera device parameter configuration request or the camera code stream parameter configuration request, the server distributes the task to the worker thread.

[0048] If the HTTP request is a GET method request, the related API interface of the camera SDK is called to obtain the relevant information and return it to the client. For example, if it is an HTTP request to obtain the camera gimbal speed, the API interface of the camera SDK related to obtaining the camera gimbal speed is called, and the camera gimbal speed is placed in the body part of the HTTP response and returned to the client after being obtained.

[0049] If the HTTP request is a request of POST method, the camera SDK related API interface is also called, and the setting result is returned to the client after the setting is successful. For example, if the HTTP request is to set the speed of the camera holder, the camera SDK API interface related to setting the speed of the camera holder is called, and the set speed of the camera holder is placed in the body part of the HTTP response and returned to the client after the setting is successful.

[0050] 4) If the HTTP request is the URL corresponding to the camera SDK upgrade preprocessing request, the server distributes the task to the upgrade thread, and then completes the upgrade function according to the following sub-steps:

[0051] a) In the upgrade thread, the upgrade preparation identification field (for example, Upgrade: Ready) is placed in the body part of the HTTP response, and the complete HTTP response is returned to the client by circularly calling send, so as to inform the client that the server is ready to receive the upgrade package and maintain the long connection.

[0052] b) The client receives the response returned in step a), in order to ensure the success rate of the upgrade, each time a preset length of data segment of the upgrade package file is read, in this embodiment, the length of the upgrade package file is 32 KB each time, the read data segment is placed in the request body of the HTTP request, the Content-Length field is added in the header field of the HTTP request, and the value is 32 KB, and the HTTP request is sent to the server through the POST method. In addition, the size of the sent upgrade package file needs to be accumulated for subsequent verification of whether the complete upgrade package file is successfully received by the server.

[0053] c) The server obtains the task from the request queue and parses the HTTP request, if it is the URL corresponding to the camera upgrade package sending, the task is distributed to the upgrade thread, the upgrade package file in the request body of the HTTP request is placed in the cache area, and the size of the received upgrade package file is accumulated, during which the server does not make a related HTTP response.

[0054] d) After the client sends all the contents of the upgrade package file, the number of sent bytes is filled into the request body of the HTTP request as a verification package, and is sent to the server through the POST method, for verifying whether the upgrade package file received by the server is complete.

[0055] e) The server gets the task from the request queue and parses the HTTP request, if it is a camera verification package, it verifies the corresponding URL, distributes the task to the upgrade thread, compares the number of bytes sent in the request body of the HTTP request with the number of bytes received in the server buffer area, if they are equal, the verification is passed, the upgrade API interface in the camera SDK is called, then the upgrade mode is entered, the received upgrade package file in the buffer area is written to the firmware step by step, after writing, the camera is restarted and the upgrade package file is loaded, after the camera starts, the server will put the upgrade completion identification field (for example, Upgrade: Done) in the body part of the HTTP response, and call send in a loop to return the complete HTTP response to the client, and tell the client that the server has completed the upgrade, and call close to close the current socket. If they are not equal, the verification fails, the server will put the upgrade failure identification field (for example, Upgrade: Fail) in the body part of the HTTP response, and call send in a loop to return the complete HTTP response to the client, and the client will receive the HTTP response and start the upgrade retry according to step b) until the upgrade is successful.

[0056] The above merely describes the preferred embodiments of the present application. Although the present application has been disclosed with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art can make many possible changes and modifications to the present application, or modify equivalent embodiments, without departing from the scope of the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the scope of the present application, shall still fall within the scope of protection of the present application.

Claims

1. A multifunctional real-time imaging platform, characterized in that: The platform includes a server and a client; The server is built on the Linux system, and its concurrency model is implemented through thread pool, non-blocking socket, epoll based on level triggering and Reactor event processing model; The design process of the server and client includes: (1) creating a thread pool, which includes a worker thread, a streaming thread, and an upgrade thread, and presetting a maximum capacity of worker threads for the thread pool; (2) Create a socket through socket, set it to non-blocking mode using fcntl, then bind it to the server's IP address and specified port, and call bind and listen to start listening for client connection requests; call epoll_create to create an epoll instance, use epoll_ctl to register the socket to the epoll instance, and listen for readable events on the socket to notify the main thread when a new client connection request is received; (3) In the main thread, epoll_wait is used to monitor the readable events on the socket. If a readable event is detected on the socket, accept is called to accept the new client connection request and return the new socket. The new socket is configured in non-blocking mode. epoll_ctl is used to register the new socket with the epoll instance and monitor the readable events on the new socket. When a readable event is detected on the new socket, recv is called in a loop until the complete HTTP request is read, and the completed HTTP request is added to the request queue. (4) Get the task from the request queue, parse the HTTP request, generate the corresponding HTTP response based on the specific content of the parsed HTTP request, and call send in a loop to return the complete HTTP response to the client.

2. The multifunctional real-time imaging platform according to claim 1, characterized in that: In the thread pool, the worker thread is used to process client requests. In order to ensure that the streaming function and upgrade function of the server are not affected by the worker thread, a streaming thread and an upgrade thread are created separately.

3. The multifunctional real-time imaging platform according to claim 1, characterized in that: In the thread pool, a higher thread priority is set for the streaming thread and the upgrade thread to ensure priority during scheduling.

4. The multifunctional real-time imaging platform according to claim 1, characterized in that: In step (4), if the HTTP request is the URL corresponding to the camera video stream preview request, the server distributes the task to the streaming thread, calls the streaming API interface in the camera SDK in the streaming thread, converts the RTMP video stream into an FLV video stream through the open source ZLMediaKit framework, places the FLV video stream in the main body of the HTTP response, calls send in a loop, returns the complete HTTP response to the client, and maintains a long connection to continue streaming.

5. The multifunctional real-time imaging platform according to claim 4, characterized in that: In step (4), after receiving the FLV video stream from the server, the client creates a new media extension source through MSE and associates the new media extension source with the HTML5 video element; Use flv.js to process FLV video streams. It parses FLV files, encodes them into MP4 segments, and loads the MP4 segments into a media extension source. The media extension source then pushes the MP4 segments to the associated HTML5 video element for playback.

6. The multifunctional real-time imaging platform according to claim 1, characterized in that: In step (4), if the HTTP request is a URL corresponding to a Web login request, the server distributes the task to a worker thread, in which the account and password in the request body of the HTTP request are parsed out respectively, and SQLite is used as the database; a connection is taken out from the database connection pool to access the database, and the parsed account and password are compared with the corresponding account and password in the database to verify whether they are equal, and the verification result is used as the main body of the HTTP response, and send is called in a loop to return the complete HTTP response to the client; if it is judged to be equal, the verification is passed, and the client enters the main page; if it is judged to be unequal, the verification fails, and the client stays on the login page.

7. The multifunctional real-time imaging platform according to claim 1, characterized in that: In step (4), if the HTTP request is a URL corresponding to a camera image parameter configuration request, a camera device parameter configuration request, or a camera stream parameter configuration request, the server distributes the task to the worker thread; If the HTTP request is a GET method request, call the camera SDK related API interface to obtain relevant information and return it to the client; if the HTTP request is a POST method request, call the camera SDK related API interface, and return the setting result to the client after successful setting.

8. The multifunctional real-time imaging platform according to claim 1, characterized in that: In step (4), if the HTTP request is the URL corresponding to the camera SDK upgrade pre-processing request, the server will dispatch the task to the upgrade thread, and then complete the upgrade function according to the following sub-steps: a) In the upgrade thread, the upgrade preparation identification field is placed in the body of the HTTP response, and send is called in a loop to return the complete HTTP response to the client, informing the client that the server is ready to receive the upgrade package, while maintaining a persistent connection; b) The client receives the response from step a), reads a data segment of a preset length from the upgrade package file each time, places the read data segment into the request body of an HTTP request, adds a field of the preset length to the header field of the HTTP request, and sends it to the server via the POST method; c) The server retrieves tasks from the request queue and parses the HTTP request. If it is a camera upgrade package sending the corresponding URL, it dispatches the task to the upgrade thread, places the upgrade package file in the HTTP request body into the cache, and accumulates the size of the received upgrade package files. During this period, the server does not respond to any related HTTP requests. d) After the client has sent all the contents of the upgrade package, it will enter the number of bytes sent as a verification packet in the request body of the HTTP request and send it to the server via the POST method to verify whether the server has received the upgrade package file completely; e) The server retrieves the task from the request queue and parses the HTTP request. If it is a camera verification package verifying the corresponding URL, it dispatches the task to the upgrade thread. It compares the number of bytes sent in the HTTP request body with the number of bytes received in the server buffer. If they are equal, the verification passes, and the upgrade API in the camera SDK is called to enter upgrade mode. If they are not equal, the verification fails. The server places the upgrade failure indicator field in the body of the HTTP response and calls send in a loop to return the complete HTTP response to the client. After receiving the HTTP response, the client starts the upgrade again according to step b) and retries until the upgrade is successful.

9. The multifunctional real-time imaging platform according to claim 8, characterized in that: The upgrade API interface in the camera SDK is called to enter the upgrade mode, specifically: the upgrade package files received in the cache area are gradually written to the firmware, and after writing, the camera is restarted and the upgrade package file is loaded. After the camera startup is completed, the server puts the upgrade completion identification field in the body of the HTTP response, and calls send in a loop to return the complete HTTP response to the client, informing the client that the server has completed the upgrade, and calls close to close the current socket.

Images