Method and system for improving user experience of video applications based on multi-terminal collaboration

Abstract

The present invention provides a method and system for improving the user experience of video applications based on multi-terminal collaboration. The method comprises: a master camera sends a target face image to N-1 slave cameras, establishes a connection, and transmits the video stream to a video server; the N camera terminals perform facial recognition on the target face and publish the similarity obtained from the recognition to a policy server; the policy server continuously subscribes to the facial recognition similarity results published by the N camera terminals and compares the recognition results; obtains the terminal number of the camera with the highest similarity and sends it to the master camera terminal; the master camera initiates a thread to receive the camera number with the highest current similarity issued by the policy server and selects whether to switch video stream transmission based on the number. The present invention enables multi-channel cameras to complete adaptive transmission switching during user movement, improving the user experience in terms of both latency and call interface service quality.

Description

Technical Field

[0001] The present invention belongs to the field of mobile communication technology, and in particular relates to a method and system for improving user experience of video applications based on multi-terminal collaboration. Background Art

[0002] In recent years, mobile communication networks and internet technologies have experienced rapid development. With the advent of the 5G era, network bandwidth has significantly increased, spurring the development of network services, particularly video services. Intelligent video applications are a key means of enhancing user experience. Intelligent video content recognition is already widely used in some typical application scenarios. For example, in home cameras, moving object detection and stranger warning technologies have been implemented. In education, intelligent video recognition enables smart education and innovative teaching management. In transportation, cameras installed at intersections can capture vehicle information such as license plates and vehicle models, as well as the number of passengers and other passenger information, and transmit it back to data centers for monitoring.

[0003] However, in some typical video application scenarios, users prioritize video service quality and user experience. For example, in video calls, due to the limitations of a single-camera terminal, users must maintain a distance from the terminal during a call, which restricts their movements. In single-camera terminal video application scenarios, mobile object recognition technologies are relatively mature. However, single terminals inherently have technical bottlenecks, such as the inability to observe the entire target area from a single camera, the inability to continuously track moving objects, and difficulty resolving occlusions in complex scenes. These limitations hinder single-camera terminal video applications from covering all scenarios.

[0004] In view of this, it is necessary to design a video application system and method based on multi-terminal collaboration to improve the user experience during the call and solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to enhance the user experience in video calls. It provides services to users simultaneously through N cameras, adopts face recognition technology and adaptive multi-stream transmission technology, and thus improves the user's video call experience during mobile in terms of communication delay and picture service quality.

[0006] To achieve the above objectives, the present invention provides a method for improving user experience of video applications based on multi-terminal collaboration, comprising the following steps:

[0007] Step S1: The main camera terminal sends a target face image to N-1 slave cameras;

[0008] Step S2: The main camera terminal establishes a connection with the video server and transmits its own video stream to the video server via WiFi, 5G, and Internet links through adaptive multi-stream transmission, which serves as the default connection in the initial stage. The N camera terminals simultaneously perform face recognition on the target face and publish the similarity obtained by recognition to the policy server via the MQTT protocol.

[0009] Step S3: The policy server continuously subscribes to the face recognition similarity results published by N camera terminals through the MQTT protocol, compares the recognition results, obtains the terminal number of the camera terminal with the greatest similarity, and sends the number to the main camera terminal;

[0010] Step S4: The main camera terminal starts a thread to receive the camera terminal number with the highest current similarity sent by the policy server, and chooses whether to switch video stream transmission according to the number.

[0011] A further improvement of the present invention is that step S1 further includes the following steps:

[0012] Step S1.1: The master camera terminal broadcasts its own information every 1 second. The information mainly includes the IP address and the flag string "master". Then it enters the monitoring state and sets the number of monitored slave camera terminals to N-1.

[0013] Step S1.2: After receiving the broadcast message, the N-1 slave camera terminal checks whether the broadcast message contains the identifier string "master". If so, it determines that a multi-terminal collaborative service has been initiated and responds to the master camera terminal with its own IP address, access method, and identifier string "video_x".

[0014] Step S1.3: After receiving the reply from the N-1 slave camera terminals, the master camera terminal stores the IP address and establishes a TCP connection with the N-1 slave camera terminals. The master camera terminal extracts the target face image at a predetermined interval and writes the image data into the shared memory. The master camera terminal, acting as the client of the TCP connection, creates a thread for reading the image from the shared memory and a thread for sending the image frame. Because the created two threads involve synchronization of the image data, a mutex lock is locked before reading and sending the image data. The process ends after a set time, i.e., the sending of the target face image ends.

[0015] Step S1.4: The N-1 slave camera terminals serve as the server end that establishes a TCP connection with the master camera terminal, receives the image data sent by the master camera, converts the format, and saves it locally.

[0016] A further improvement of the present invention is that the setting time in step S1.3 is 10 seconds.

[0017] A further improvement of the present invention is that step S4 further includes the following steps:

[0018] Step S4.1: The main camera terminal saves the camera terminal number of the channel with the highest similarity in the previous recognition, and compares it with the video terminal number of the channel with the highest similarity in the current recognition;

[0019] Step S4.2: If the two camera terminal numbers are different, the video server is notified to disconnect the previously established connection and re-establish the connection; if the number with the highest similarity is in the slave camera terminal, the master camera terminal pulls the video stream of the slave camera terminal and forwards it to the video server;

[0020] Step S4.3: If the camera terminal numbers are the same twice, the previously established connection with the video server is maintained.

[0021] A further improvement of the present invention is that step S2 further includes the following steps:

[0022] Step A1: Establish multiple TCP connections between the main camera terminal and the video server. The video server monitors ports 1000, 1200, and 1400, respectively. When the main camera terminal transmits via the WiFi link by default, the main camera terminal uses iptables -t mangle to add different tags to data packets sent to ports 1200 and 1400 of the video server. Data packets matching the rules are added to routing tables 8 and 9, respectively. Routing tables 8 and 9 are set to use 5G transmission and Internet transmission by default, respectively.

[0023] Step A2: The main camera terminal distributes the pulled video stream data to multiple sub-stream transmissions between the main camera terminal and the video server. By default, transmission is only carried out through the WiFi link. During the transmission process, a thread is created to monitor the current WiFi link transmission quality. When the transmission quality of the WiFi link is poor, 5G link transmission is added. When the thread monitors that the WiFi link meets normal transmission conditions, the data packet distribution strategy is changed back to transmission only on the WiFi link.

[0024] A further improvement of the present invention is that the ratio of data packets distributed between the two links is set to 1:1.

[0025] A further improvement of the present invention is that, in step S2, N camera terminals simultaneously perform face recognition on the target face at a frequency of 1 time per second.

[0026] A further improvement of the present invention is that the predetermined time is 1 second, that is, the main camera terminal extracts an image of the target face every 1 second.

[0027] A further improvement of the present invention is that the video server is composed of N-1 slave camera terminals.

[0028] To achieve the above object of the invention, the present invention also provides a system for improving user experience of video applications based on multi-terminal collaboration, which can implement any of the methods described above.

[0029] The beneficial effects of the present invention are as follows: the present invention uses N cameras to simultaneously provide services to users, face recognition technology and adaptive multi-stream transmission technology, thereby improving the user's video call experience during mobile in terms of communication delay and picture service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an architecture diagram of a system and method for improving user experience of video applications based on multi-terminal collaboration provided by the present invention.

[0031] Figure 2 This is a schematic diagram of terminal device discovery provided by the present invention.

[0032] Figure 3 This is a flow chart of the main camera group sending and identifying target pictures provided by the present invention.

[0033] Figure 4 This is a flow chart of the main camera switching and transmitting video stream provided by the present invention.

[0034] Figure 5 This is a flow chart of adaptive multi-stream transmission in heterogeneous networks provided by the present invention. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] It should be emphasized that, in the process of describing the present invention, various formulas and constraints are distinguished by using consistent labels, but it is not excluded that different labels are used to mark the same formulas and / or constraints. The purpose of this setting is to more clearly illustrate the characteristics of the present invention.

[0037] like Figure 1As shown, the present invention provides a system and method for improving the user experience of video applications based on multi-terminal collaboration. The system includes N-way camera terminal devices, a video server and a policy server, and the N-way camera terminal devices are composed of one main camera and N-1 slave cameras. The N-way cameras are accessed by wired or wireless (for example: WiFi), wherein the main camera supports 5G and Internet public network access, and serves as the main device for interacting with the video server. The main camera terminal sends the target face picture to the N-1 slave cameras, and after each slave camera terminal obtains the target face picture, it starts face recognition. The recognition result is uploaded to the policy server every 1 second, and the policy server sends instructions to the main camera according to the result of each recognition. The main camera pulls the camera video stream with the best experience effect, and transmits it through adaptive multi-stream transmission through WiFi, 5G, and Internet links, so that multiple cameras can complete adaptive transmission switching during the movement of people.

[0038] like Figures 2 to 5 As shown, the present invention provides a method for improving user experience of video applications based on multi-terminal collaboration, which mainly includes the following steps:

[0039] Step S1, the main camera terminal sends the target face picture to the N-1 slave camera groups; Step S1 also includes the N-1 slave camera terminal device discovery process and the main camera terminal device sending the target picture process.

[0040] In step S2, the main camera terminal establishes a connection with the video server and transmits its own video stream to the video server via WiFi, 5G, and Internet links through adaptive multi-stream transmission, which serves as the default connection in the initial stage. The N camera terminals simultaneously perform face recognition on the target face and publish the similarity obtained by recognition to the policy server via the MQTT protocol.

[0041] Step S3: The policy server continuously subscribes to the face recognition similarity results published by N camera terminals through the MQTT protocol, and compares the recognition results; obtains the terminal number of the camera with the greatest similarity, and sends the number to the main camera terminal.

[0042] Step S4: The main camera terminal starts a thread to receive the camera terminal number with the highest current similarity sent by the policy server, and chooses whether to switch video stream transmission according to the number.

[0043] The method of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] The N-1-way slave camera terminal device discovery process of the present invention is as follows Figure 2 As shown:

[0045] Step S1.1: The master camera broadcasts its information every second, including its IP address and the symbol "master". It then enters a listening state and sets the number of slave cameras to be monitored to N-1.

[0046] In step S1.2, after receiving the broadcast message, the N-1 slave camera checks whether the broadcast message contains the identifier string "master". If so, it determines that a multi-terminal collaborative service has been initiated and responds to the master camera with its own IP address, access method, and identifier string "video_x".

[0047] The process of sending target pictures by the main camera device of the present invention is as follows: Figure 3 As shown:

[0048] Step S1.3: After receiving the reply from the N-1 slave cameras, the master camera stores the IP address and establishes a TCP connection with the N-1 slave cameras. The master camera extracts the target face image every 1 second and writes the image data to the shared memory. The master camera, acting as the client of the TCP connection, creates thread 1 to read the image from the shared memory and thread 2 to send the image frame. Because the two threads involved in the synchronization of the image data, a mutex is locked before reading and sending the image data. The process ends after 10 seconds, ending the sending of the target face image.

[0049] Step S1.4: The N-1 slave cameras serve as the server end that establishes a TCP connection with the master camera, receives the image data sent by the master camera, converts the format, and saves it locally.

[0050] The specific steps involved in the video stream adaptive transmission switching process of the present invention, namely step S4, are as follows: Figure 4 As shown:

[0051] Step S4.1: The main camera saves the camera number of the previous camera with the highest similarity, and compares it with the video number of the current camera with the highest similarity;

[0052] Step S4.2: If the camera numbers are different, the video server is notified to disconnect the previously established connection and re-establish the connection; if the number with the highest similarity is among the slave cameras, the master camera pulls the video stream of the slave camera and forwards it to the video server;

[0053] Step S4.3: If the camera numbers are the same twice, the previously established connection with the video server is maintained.

[0054] In step S2, multiple transmission channels such as WiFi, 5G, and Internet are established between the main camera and the video server to perform adaptive multi-stream transmission. Figure 5 The specific steps are as follows:

[0055] Step A1: The main camera and the video server establish multiple TCP connections. The video server monitors ports 1000, 1200, and 1400, respectively. When the main camera transmits data via the WiFi link by default, the main camera uses iptables -t mangle to add different tags to data packets sent to ports 1200 and 1400 of the video server. Data packets that match the rules are added to routing tables 8 and 9, respectively. Routing table 8 defaults to 5G transmission, and routing table 9 defaults to Internet transmission.

[0056] Step A2: The main camera distributes the pulled video stream data to multiple sub-stream transmission channels between the main camera and the video server. By default, transmission is performed only over the WiFi link. During the transmission process, a thread is created to monitor the transmission quality of the current WiFi link. If the transmission quality of the WiFi link is poor, a 5G link is added for transmission, and the ratio of data packets distributed between the two links is set to 1:1. If the thread monitors that the WiFi link meets normal transmission conditions, the data packet distribution strategy is changed back to transmission only over the WiFi link.

[0057] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for improving user experience of video applications based on multi-terminal collaboration, characterized by: The following steps are involved: Step S1: The main camera terminal sends a target face image to N-1 slave cameras; Step S2: The main camera terminal establishes a connection with the video server and transmits its own video stream to the video server via WiFi, 5G, and Internet links through adaptive multi-stream transmission, which serves as the default connection in the initial stage. The N camera terminals simultaneously perform face recognition on the target face and publish the similarity obtained by recognition to the policy server via the MQTT protocol. Step S3: The policy server continuously subscribes to the face recognition similarity results published by N camera terminals through the MQTT protocol, compares the recognition results, obtains the terminal number of the camera terminal with the greatest similarity, and sends the number to the main camera terminal; Step S4: The main camera terminal starts a thread to receive the camera terminal number with the highest current similarity sent by the policy server, and chooses whether to switch video stream transmission based on the number; Step S1 further includes the following steps: Step S1.1: The master camera terminal broadcasts its own information every 1 second. The information mainly includes the IP address and the flag string "master". Then it enters the monitoring state and sets the number of monitored slave camera terminals to N-1. Step S1.2: After receiving the broadcast message, the N-1 slave camera terminal checks whether the broadcast message contains the identifier string "master". If so, it determines that a multi-terminal collaborative service has been initiated and responds to the master camera terminal with its own IP address, access method, and identifier string "video_x". Step S1.3: After receiving the reply from the N-1 slave camera terminals, the master camera terminal stores the IP address and establishes a TCP connection with the N-1 slave camera terminals. The master camera terminal extracts the target face image at a predetermined interval and writes the image data into the shared memory. The master camera terminal, acting as the client of the TCP connection, creates a thread for reading the image from the shared memory and a thread for sending the image frame. Because the created two threads involve synchronization of the image data, a mutex lock is locked before reading and sending the image data. The process ends after a set time, i.e., the sending of the target face image ends. Step S1.4: The N-1 slave camera terminals serve as the server end that establishes a TCP connection with the master camera terminal, receives the image data sent by the master camera, converts the format, and saves it locally.

2. The method according to claim 1, wherein: The set time in step S1.3 is 10 seconds.

3. The method according to claim 1, wherein: Step S4 further includes the following steps: Step S4.1: The main camera terminal saves the camera terminal number of the channel with the highest similarity in the previous recognition, and compares it with the video terminal number of the channel with the highest similarity in the current recognition; Step S4.2: If the two camera terminal numbers are different, the video server is notified to disconnect the previously established connection and re-establish the connection; if the number with the highest similarity is in the slave camera terminal, the master camera terminal pulls the video stream of the slave camera terminal and forwards it to the video server; Step S4.3: If the camera terminal numbers are the same twice, the previously established connection with the video server is maintained.

4. The method according to claim 3, wherein: Step S2 further includes the following steps: Step A1: Establish multiple TCP connections between the main camera terminal and the video server. The video server monitors ports 1000, 1200, and 1400, respectively. When the main camera terminal transmits via the WiFi link by default, the main camera terminal uses iptables -t mangle to add different tags to data packets sent to ports 1200 and 1400 of the video server. Data packets matching the rules are added to routing tables 8 and 9, respectively. Routing tables 8 and 9 are set to use 5G transmission and Internet transmission by default, respectively. Step A2: The main camera terminal distributes the pulled video stream data to multiple sub-stream transmissions between the main camera terminal and the video server. By default, transmission is only carried out through the WiFi link. During the transmission process, a thread is created to monitor the current WiFi link transmission quality. When the transmission quality of the WiFi link is poor, 5G link transmission is added. When the thread monitors that the WiFi link meets normal transmission conditions, the data packet distribution strategy is changed back to transmission only on the WiFi link.

5. The method according to claim 4, characterized in that: The ratio of data packets distributed between the two links is set to 1:

1.

6. The method according to claim 5, characterized in that: In step S2, N camera terminals simultaneously perform face recognition on the target face at a frequency of 1 time per second.

7. The method according to claim 1, wherein: The predetermined time is 1 second, that is, the main camera terminal extracts the image of the target face every 1 second.

8. The method according to claim 1, wherein: The video server is composed of N-1 slave camera terminals.

9. A system for improving user experience of video applications based on multi-terminal collaboration, the system being capable of implementing the method according to any one of claims 1 to 8.

Images