An intelligent basketball live broadcast system based on human-computer cooperation

Abstract

The application discloses a kind of intelligent basketball live broadcast systems based on man-machine cooperation, belong to automatic live broadcast technical field.It includes: mobile terminal device, two-dimensional holder and PC end management system;Mobile terminal device is used to real-time acquisition ball game picture, and obtains live broadcast focus coordinates by target detection to ball game picture;Two-dimensional holder is used to change control signal pulse width based on live broadcast focus coordinates, drive PWM steering gear to correct ball game picture collected by mobile terminal device by microcontroller;PC end management system is used to receive ball game picture collected by mobile terminal device, and upload to server end for user to pull stream and watch.The application can realize the intelligent live broadcast of basketball event, and the identification process is accurate and efficient, flexible to use and low in cost, with wide application prospect and commercial value.

Description

Technical Field

[0001] This invention relates to the field of automatic live streaming technology, and more specifically to an intelligent basketball live streaming system based on human-machine collaboration. Background Technology

[0002] With the rapid development of the internet and live streaming technology, more and more users are watching live sports events through online streaming platforms. Among numerous sports events, basketball has become a mainstream event enjoyed by users of all ages. To ensure a good viewing experience, companies have to invest more manpower and resources in live streaming, using manual camera operation. Even so, manual operation still has many shortcomings, such as technical errors caused by cameramen during long live shooting sessions and a poor viewing experience due to the slow turning of manually operated cameras.

[0003] Currently, mature live streaming equipment and data systems are generally used in professional sports events due to their high cost. These systems employ dedicated cameramen to control camera movement and shooting angles to capture exciting moments and transmit these images to viewers in real time. Mainstream automated live streaming systems generally fall into two categories: one deploys a tracking module in the equipment to manually select targets for shooting, ensuring the selected targets do not move too fast, the surrounding environment is not too complex, and the lost targets cannot be retrieved; the other uses panoramic cameras, which obtain candidate foreground targets by adaptively modeling the panoramic video background, fusing and processing these candidates to form candidate target points, and employing dynamic trajectory management to achieve stable multi-target tracking in the panoramic video. However, panoramic cameras are expensive and not suitable for mass live streaming.

[0004] Therefore, how to achieve automatic collection, identification, and rebroadcasting in live streaming, improve usage flexibility, reduce costs, and enhance application prospects and commercial value are problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides an intelligent basketball live streaming system based on human-machine collaboration, so as to at least solve some of the technical problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides an intelligent basketball live streaming system based on human-machine collaboration, comprising: a mobile device, a two-dimensional gimbal, and a PC-based management system;

[0008] The mobile device is used to capture the game footage in real time and to obtain the coordinates of the live focus by performing target detection on the game footage.

[0009] The two-dimensional gimbal is used to drive a PWM servo motor to correct the game footage captured by the mobile device by changing the control signal pulse width through a microcontroller based on the live broadcast focus coordinates.

[0010] The PC-based management system is used to receive the game footage captured by the mobile device and upload it to the server for users to stream and watch.

[0011] Furthermore, the mobile device is equipped with a data acquisition module and a target detection module.

[0012] The data acquisition module is used to acquire the game footage in real time and transmit it to the PC-side management system using the RTSP protocol in a local area network environment.

[0013] The target detection module is used to perform target detection on the game footage acquired by the data acquisition module, obtain the live focus coordinates, and transmit the live focus coordinates to the two-dimensional pan-tilt unit.

[0014] Furthermore, the target detection module includes:

[0015] The detection box generation submodule is used to define the court area from the game screen and call a self-trained YOLOv5 model to perform target recognition and detection of players and the ball in the court area, generating a detection box corresponding to each target; represented as:

[0016] boundingBox = [x, y, w, h]

[0017] Where boundingBox represents the detection box; x represents the horizontal coordinate of the upper left position of the detection box; y represents the vertical coordinate of the upper left position of the detection box; w represents the width of the detection box; h represents the height of the detection box; and the units of x, y, w, and h are all pixels.

[0018] The target coordinate generation submodule is used to select the center point of each detection box as the corresponding target coordinate. Specifically, it uses the sum of the x-coordinate of the upper left position of the detection box and the midpoint of the detection box width as the target x-coordinate, and the sum of the y-coordinate of the upper left position of the detection box and the detection box height as the target y-coordinate, generating the corresponding target coordinates; expressed as:

[0019] targetPos = ((x + 0.5w), (y + h))

[0020] Where targetPos represents the target coordinates; x+0.5w represents the x-coordinate of the target coordinates; and y+h represents the y-coordinate of the target coordinates.

[0021] The focus coordinate generation submodule is used to fit the minimum circumcircle coordinates based on the player target coordinates and ball target coordinates in the target coordinates, and use the center coordinates of the circle as the live focus coordinates.

[0022] Furthermore, the focus coordinate generation submodule includes:

[0023] The circumscribed matrix determination unit is used to filter out the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate from the ball target coordinates and the player target coordinates, respectively, and generate the circumscribed matrix;

[0024] An initial circle creation unit is used to create an initial circle at the center of the outer matrix; the radius of the initial circle is half the width of the outer matrix.

[0025] The circle radius update unit is used to calculate the distance between each target coordinate and the current circle center. If the distance is greater than the current circle radius, the circle radius is updated.

[0026] The focus coordinate generation unit is used to take the circle obtained after traversing all target coordinates as the smallest circumcircle, and take the center coordinates of the smallest circumcircle as the live focus coordinates.

[0027] Furthermore, the microcontroller of the two-dimensional gimbal is equipped with a servo motor drive module;

[0028] The servo drive module is used to calculate the offset between the live focus coordinates and the center of the game screen on the x and y axes, and to change the control signal pulse width based on the offset to drive the PWM servo to correct the game screen so that the live focus coordinates are closer to the game screen.

[0029] Furthermore, the microcontroller of the two-dimensional gimbal is equipped with a gear switching module;

[0030] The gear shifting module is used to switch between automatic and manual gears by verifying the switching flag in the servo drive command.

[0031] Furthermore, the PC-based management system is equipped with a live streaming module and a collaborative control module.

[0032] The live streaming module receives the game footage transmitted from the mobile device via the RTSP protocol and displays it in real time on the PC. At the same time, it uploads the game footage to the server for users to stream and watch.

[0033] The collaborative control module is used to locate and adjust the game footage by adjusting the servo motor angle increment when the live streaming module fails to receive the game footage.

[0034] Furthermore, the servo angle increment is expressed as:

[0035] delta_degree = Kp * Offset

[0036] Where delta_degree represents the servo angle increment; Kp represents the proportional coefficient; and Offset represents the offset.

[0037] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a framework for an intelligent basketball live streaming system based on human-machine collaboration, which has the following beneficial effects:

[0038] This invention first uses a mobile camera mounted on a gimbal to capture footage of a basketball court, generating the coordinates of the current live stream focus based on target detection. The mobile device then wirelessly transmits these coordinates to a microcontroller, which uses a PID algorithm to control the gimbal to automatically track the focus. Simultaneously, the mobile device's feed is transmitted to a PC via a local area network for streaming. For the rare instances of tracking failure, manual correction is possible, enabling human-machine collaboration. This invention enables intelligent live streaming of basketball games, offering accurate and efficient recognition, flexible use, and low cost, thus possessing broad application prospects and commercial value.

[0039] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the framework of an intelligent basketball live streaming system based on human-machine collaboration, provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of real-time image target detection for basketballs and players provided in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the circumcircle of the focal point formed after outlier filtering, provided in an embodiment of the present invention.

[0044] Figure 4 This is a schematic diagram of the external shape of a two-dimensional gimbal structure provided in an embodiment of the present invention.

[0045] Figure 5A schematic diagram of the PC-side management system interface provided in an embodiment of the present invention.

[0046] Figure 6 This is an example diagram of a user streaming interface provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] This invention discloses an intelligent basketball live streaming system based on human-machine collaboration, such as... Figure 1 As shown, it includes: a mobile device, a 2D gimbal, and a PC-based management system; wherein:

[0049] Mobile devices are used to capture real-time footage of the game and to obtain the coordinates of the live stream focus by performing object detection on the game footage.

[0050] A two-dimensional gimbal is used to correct the game footage captured by a mobile device by changing the pulse width of the control signal through a microcontroller based on the live focus coordinates.

[0051] The PC-based management system is used to receive game footage captured by mobile devices and upload it to the server for users to stream and watch.

[0052] Next, we will provide a detailed explanation of each of the above parts.

[0053] 1. Mobile devices:

[0054] The mobile device is mounted on a cloud platform and contains a data acquisition module and a target detection module; among which:

[0055] (1) Data acquisition module:

[0056] This data acquisition module is used to call the mobile camera to capture the game footage in real time, and transmit it to the PC management system using the RTSP protocol in a local area network environment; and transmit the image information to be identified to the target detection module.

[0057] (2) Target detection module:

[0058] This target detection module is used to perform target detection on the game footage captured by the data acquisition module, obtain the live focus coordinates, and transmit the live focus coordinates to the 2D pan-tilt unit. Specifically, the target detection module includes a detection box generation submodule, a target coordinate generation submodule, and a focus coordinate generation submodule; wherein:

[0059] 1) Detection box generation submodule:

[0060] This detection box generation submodule is used to define the field area from the game footage and call a self-trained YOLOv5 model to perform target recognition and detection of players and the ball within the field area, generating a detection box corresponding to each target (player target or ball target); represented as:

[0061] boundingBox = [x, y, w, h]

[0062] Where boundingBox represents the detection box; x represents the horizontal coordinate of the upper left position of the detection box; y represents the vertical coordinate of the upper left position of the detection box; w represents the width of the detection box; h represents the height of the detection box; and the units of x, y, w, and h are all pixels.

[0063] 2) Target coordinate generation submodule:

[0064] This target coordinate generation submodule is used to select the center point of each detection box as the corresponding target coordinate. Specifically, it uses the sum of the x-coordinate of the upper left position of the detection box and the midpoint of the detection box width as the target x-coordinate, and the sum of the y-coordinate of the upper left position of the detection box and the detection box height as the target y-coordinate, thus generating the corresponding target coordinates. Figure 2 As shown; represented as:

[0065] targetPos = ((x + 0.5w), (y + h))

[0066] Where targetPos represents the target coordinates; x+0.5w represents the x-coordinate of the target coordinates; and y+h represents the y-coordinate of the target coordinates.

[0067] 3) Focus coordinate generation submodule:

[0068] In actual basketball games, players are often irregularly distributed. To make the focal point more suitable for live viewing, outlier filtering is performed on the obtained player coordinates, and basketball coordinates are added to the filtered coordinate set. The distribution of the new coordinate set is then fitted using the minimum circumcircle, and the center coordinates of the circle are the focal coordinates. Therefore, this embodiment of the invention includes a focal coordinate generation submodule. This focal coordinate generation submodule is used to fit the minimum circumcircle based on the player target coordinates and ball target coordinates in the target coordinates, and to use the center coordinates of the circle as the live focus coordinates. Specifically, this focal coordinate generation submodule includes a circumcircle matrix determination unit, an initial circle creation unit, a circle radius update unit, and a focal coordinate generation unit; wherein:

[0069] ① The outer matrix determination unit is used to select the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate from the ball target coordinates and player target coordinates respectively, and generate the outer matrix;

[0070] ② Initial circle creation unit, used to create an initial circle at the center of the outer matrix; the radius of the initial circle is half the width of the outer matrix;

[0071] ③ The circle radius update unit is used to calculate the distance between each target coordinate and the current circle center, that is, to calculate the distance between each ball target coordinate and the initial circle center mentioned above, and to calculate the distance between each player target coordinate and the initial circle center mentioned above; if the distance is greater than the current circle radius, then the circle radius is updated;

[0072] ④ The focus coordinate generation unit is used to take the circle obtained after traversing all target coordinates as the smallest circumcircle, and the center coordinates of the smallest circumcircle as the live focus coordinates, such as... Figure 3 As shown;

[0073] The coordinates of the center of the smallest circumcircle are represented as (center_x, center_y), and the radius is represented as radius.

[0074] Where center_x and center_y are the average of the maximum and minimum x-coordinates of the circumscribed rectangle, respectively:

[0075] center_x = (max_x + min_x) / 2

[0076] center_y = (max_y + min_y) / 2

[0077] The calculation process for the radius can be expressed as follows:

[0078] radius=max(max_x-min_x,max_y-min_y) / 2

[0079] Where max_x represents the maximum value of the x-coordinate; min_x represents the minimum value of the x-coordinate; max_y represents the maximum value of the y-coordinate; and min_y represents the minimum value of the y-coordinate.

[0080] 2. Two-dimensional gimbal:

[0081] This two-dimensional gimbal consists of a microcontroller and two PWM-driven servo motors, enabling rotation in two dimensions, such as... Figure 4 As shown; in practical implementation, the mobile device needs to be mounted on the bracket above the gimbal, and the servo motor is connected to the microcontroller below. The microcontroller is equipped with two wireless communication modules (Bluetooth or wireless), which are named Bluetooth1 / Wireless1 and Bluetooth2 / Wireless2 here. Turn on Bluetooth / Wireless on the mobile device and connect it to Bluetooth1 / Wireless1 on the microcontroller. After ensuring normal communication, open the mobile app to automatically access the device's camera to identify the basketball and the player, calculate the live focus coordinates and send them to the microcontroller via wireless communication. At the same time, use the preset RTSP address to transmit the captured real-time image to the PC management system.

[0082] Furthermore, the microcontroller of this two-dimensional gimbal is equipped with a servo drive module and a gear shifting module; wherein:

[0083] (1) Servo drive module:

[0084] This servo drive module is used to calculate the offset between the live broadcast focus coordinates and the center of the game screen on the x and y axes, and to change the control signal pulse width based on the offset to drive the PWM servo to correct the game screen so that the live broadcast focus coordinates are closer to the game screen.

[0085] Specifically, the rotation angles of the upper and lower arms of the two-degree-of-freedom gimbal are controlled by a PID algorithm, so that the focus is as close as possible to the center of the ball game screen.

[0086] If we define the offset as Offset, then the specific formula for calculating Offset is:

[0087] Offset = realPos - targetPos

[0088] Where realPos represents the center coordinate of the game screen; taking the x-axis as an example, if the screen resolution is 800*600, then realPos.x is 400, and targetPos represents the live focus coordinate; in practical applications, considering different resolution factors, the offset is scaled to [-1,1] as a whole, and the rotation direction of the servo angle can be obtained by the sign of Offset.

[0089] (2) Gear shifting module:

[0090] This gear shifting module is used to switch between automatic and manual gears by verifying the switching flag in the servo drive command.

[0091] The servo drive command is represented as follows:

[0092] ctrlCmd = [x,y,auto,switch,E]

[0093] Where x and y represent the horizontal and vertical coordinates of the live stream focus, respectively; auto represents the automatic / manual flag, 1 for automatic and 0 for manual; switch represents the switching flag, 1 for switching and 0 for not switching; E represents the command end identifier. Taking manual mode as an example: if the received command is "130,120,1,0E", the auto check bit is 1, indicating that this command comes from automatic mode and will not be executed; otherwise, the command will be received and executed.

[0094] 3. PC-based management system:

[0095] The interface diagram of the PC-based management system is shown below. Figure 5 As shown, the PC-based management system includes a live streaming module and a collaborative control module; among which:

[0096] (1) Live streaming module

[0097] This live streaming module receives the game footage transmitted from mobile devices via the RTSP protocol and displays it in real time on the PC. Simultaneously, it uploads the game footage to the server for users to stream and watch. Figure 6 As shown;

[0098] (2) Cooperative control module:

[0099] This collaborative control module is used to locate and adjust the game footage by adjusting the servo motor angle increment when the live streaming module fails to receive the game footage.

[0100] The servo angle increment is expressed as:

[0101] delta_degree = Kp * Offset

[0102] Here, delta_degree represents the servo angle increment; Kp represents the proportional coefficient; and Offset represents the offset. The value of Kp varies depending on the servo. Generally, if the servo swing amplitude is too large, it indicates that the proportional system is too large and Kp needs to be reduced; conversely, Kp needs to be increased. In addition, to avoid small servo swings caused by the focus being near the center of the image, the "dead zone" range Φ needs to be set by comprehensively considering the range of servo rotation angle and the image resolution. The "dead zone" here refers to a range value that is related to the actual hardware, shooting height, and angle. When the offset is less than the "dead zone" Φ, the servo does not rotate, thus solving the problem of jitter near the target value. If the dead zone is not involved, the servo will rotate if the focus position changes slightly, causing jitter near the target value.

[0103] In the implementation of the PC-based management system, the PC's Bluetooth / wireless connection to the microcontroller's Bluetooth 2 / wireless 2 must be ensured to guarantee normal communication. Simultaneously, clicking the "Play" button on the PC displays the video information captured from the mobile device. When switching between manual and automatic modes, first click the "Shift" button to select the switching module, and then click the "Lock" button to complete the switch. The PC-based management system includes a collaborative control module and a live streaming module. When tracking is successful, the system continuously locks onto and tracks the target using algorithms. If tracking fails, manual correction can be performed through the collaborative control module; the operator can directly specify the angle increment using the "±X°" button on the interface to quickly complete positioning or fine-tuning. The live streaming module receives real-time video transmitted from the mobile device via the RTSP protocol and displays it on the PC in real time, while simultaneously uploading the video to a cloud server for users to stream and watch.

[0104] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0105] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A human-machine collaborative intelligent basketball live streaming system, characterized in that, include: Mobile device, 2D gimbal, and PC management system; The mobile device is used to capture the game footage in real time and to obtain the coordinates of the live focus by performing target detection on the game footage. The two-dimensional gimbal is used to drive a PWM servo motor to correct the game footage captured by the mobile device by changing the control signal pulse width through a microcontroller based on the live broadcast focus coordinates. The PC-based management system is used to receive the game footage captured by the mobile device and upload it to the server for users to stream and watch. The mobile device is equipped with a data acquisition module and a target detection module. The data acquisition module is used to acquire the game footage in real time and transmit it to the PC-side management system using the RTSP protocol in a local area network environment. The target detection module is used to perform target detection on the game footage acquired by the data acquisition module to obtain the coordinates of the live focus. And transmit the live stream focus coordinates to the two-dimensional gimbal; The target detection module includes: The detection box generation submodule is used to define the field area from the game screen and call a self-trained YOLOv5 model to perform target recognition and detection of players and the ball in the field area, generating a detection box corresponding to each target; represented as: in, boundingBox Represents the detection box; where x The x-coordinate of the upper left position of the detection box; y The vertical coordinate represents the position of the upper left corner of the detection box; w Indicates the width of the detection frame; h Indicates the height of the detection frame; x , y , w and h The units are all pixels; The target coordinate generation submodule is used to select the center point of each detection box as the corresponding target coordinate. Specifically, it uses the sum of the x-coordinate of the upper left position of the detection box and the midpoint of the detection box width as the target x-coordinate, and the sum of the y-coordinate of the upper left position of the detection box and the detection box height as the target y-coordinate, generating the corresponding target coordinates; expressed as: in, targetPos Indicate the target coordinates; x +0.5 w The x-coordinate of the target coordinates; y + h The ordinate represents the target coordinates; The focus coordinate generation submodule is used to fit the minimum circumcircle coordinates based on the player target coordinates and ball target coordinates in the target coordinates, and use the center coordinates of the circle as the live focus coordinates. The focus coordinate generation submodule includes: The circumscribed matrix determination unit is used to filter out the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate from the ball target coordinates and the player target coordinates, respectively, and generate the circumscribed matrix; An initial circle creation unit is used to create an initial circle at the center of the outer matrix; the radius of the initial circle is half the width of the outer matrix. The circle radius update unit is used to calculate the distance between each target coordinate and the current circle center. If the distance is greater than the current circle radius, the circle radius is updated. The focus coordinate generation unit is used to take the circle obtained after traversing all target coordinates as the smallest circumcircle, and take the center coordinates of the smallest circumcircle as the live focus coordinates.

2. The intelligent basketball live streaming system based on human-machine collaboration according to claim 1, characterized in that, The microcontroller of the two-dimensional gimbal is equipped with a servo motor drive module. The servo drive module is used to calculate the offset between the live focus coordinates and the center of the game screen on the x and y axes, and to change the control signal pulse width based on the offset to drive the PWM servo to correct the game screen so that the live focus coordinates are closer to the game screen.

3. The intelligent basketball live streaming system based on human-machine collaboration according to claim 1, characterized in that, The microcontroller of the two-dimensional gimbal has a gear switching module deployed in it; The gear shifting module is used to switch between automatic and manual gears by verifying the switching flag bit in the servo drive command.

4. The intelligent basketball live streaming system based on human-machine collaboration according to claim 2, characterized in that, The PC-based management system includes a live streaming module and a collaborative control module. The live streaming module receives the game footage transmitted from the mobile device via the RTSP protocol and displays it in real time on the PC. At the same time, it uploads the game footage to the server for users to stream and watch. The collaborative control module is used to locate and adjust the game footage by adjusting the servo motor angle increment when the live streaming module fails to receive the game footage.

5. The intelligent basketball live streaming system based on human-machine collaboration according to claim 4, characterized in that, The servo angle increment is expressed as: in, delta _ degree Indicates the increment of the servo angle; Kp _ represents the scaling factor; _Offset_ represents the offset.

Images