A method and apparatus for processing images of a drone fleet, and a user terminal

Abstract

The application provides a method and device for processing images of a drone formation, a readable storage medium and a user terminal. The method comprises: acquiring an image of a drone formation taken by a user terminal; analyzing the image of the drone formation and matching the analysis result with a preset drone formation library; and triggering a preset processing action according to the matching result. The method provided by the application can enhance the interaction between a user and a drone flight show and provide better user experience.

Description

Technical Field

[0001] This invention relates to the field of drone applications, and in particular to a method, apparatus, readable storage medium, and user terminal for processing drone queuing images. Background Technology

[0002] With the development of drone technology, drones are being widely used in drone shows. Current drone shows typically involve multiple drones forming a formation to showcase various drone formations to the audience. While this style of performance can deliver spectacular visual effects, it lacks interactivity. Summary of the Invention

[0003] Therefore, the present invention provides a method, apparatus, readable storage medium, and user terminal for processing images of drone queuing, in an attempt to solve or at least alleviate at least one of the problems mentioned above.

[0004] According to one aspect of the present invention, a method for processing images of drone queues is provided, comprising:

[0005] Acquire images of drone queues captured by the user terminal;

[0006] The drone queue image is analyzed, and the analysis results are matched with a preset drone queue database.

[0007] Based on the matching results, a preset processing action is triggered.

[0008] Optionally, parsing the drone queue image and matching the parsing results with a preset drone queue database includes:

[0009] Analyze the feature point information of the drone queue image;

[0010] Calculate the similarity between the feature point information of the drone queue image and multiple sets of feature point information in a preset drone queue library, and determine the set of feature point information with the highest similarity from the multiple sets of feature point information;

[0011] The matching result is determined based on the set of feature points with the highest similarity.

[0012] The highest value of the similarity is higher than a preset threshold.

[0013] Optionally, after parsing the feature point information of the drone queuing image, the method further includes:

[0014] Determine preset target feature points from the feature points of the drone queue image;

[0015] Determine the transformation matrix of the feature points of the UAV queue image based on the target feature points;

[0016] The feature point information of the drone queue image is transformed according to the transformation matrix.

[0017] Optionally, parsing the drone queue image and matching the parsing results with a preset drone queue database includes:

[0018] The drones in the drone queue image are detected by a pre-trained target detection network.

[0019] Based on the detection information from the target detection network, the coordinate information of each drone in the drone queue image is determined;

[0020] The coordinate information of each drone in the drone queue image is matched with multiple sets of coordinate information in a preset drone queue library.

[0021] Optionally, the coordinate information of each drone in the drone queue image is matched with multiple sets of coordinate information in a preset drone queue database, including:

[0022] The coordinate information of each drone in the drone queue image is matched with multiple coordinate ranges included in each set of coordinate information in the preset drone queue library.

[0023] Optionally, before matching the coordinate information of each drone in the drone queue image with multiple sets of coordinate information in a preset drone queue database, the following steps are included:

[0024] Calculate the length or width of the drone queue based on the coordinate information of each drone in the drone queue image;

[0025] Based on the length or width of the drone queue and the length or width corresponding to multiple sets of coordinate information in the preset drone queue library, determine the adjustment ratio of the coordinate information of the drone queue.

[0026] Adjust the coordinate information of each drone in the drone queue image according to the adjustment ratio of the drone queue coordinate information.

[0027] Optionally, before matching the coordinate information of each drone in the drone queue image with multiple sets of coordinate information in a preset drone queue database, the following steps are included:

[0028] Acquire the positioning data of the drone queue, the sensing data of the three-axis accelerometer of the user terminal, and the positioning data of the user terminal;

[0029] Based on the positioning data of the drone queue, the sensing data of the three-axis accelerometer of the user terminal, and the positioning data of the user terminal, the shooting position of the user terminal relative to the drone queue is determined.

[0030] The coordinate information of each drone in the drone queue image is adjusted according to the shooting position of the user terminal relative to the drone queue.

[0031] Optionally, triggering a preset processing action based on the matching result includes:

[0032] After a successful match, obtain the matching information from the drone queue library;

[0033] Based on the matching information, a preset processing action is triggered.

[0034] Optionally, the preset processing actions include:

[0035] Load the preset links in the user interface; or,

[0036] Load a preset game into the user interface; or,

[0037] Load the red envelope animation in the user interface; or,

[0038] Load product animations in the user interface; or,

[0039] Display a preset pattern on the user interface; or,

[0040] Display preset text in the user interface; or,

[0041] The user interface displays the drone control interface; wherein the drone control interface includes interactive buttons for drone formation changes.

[0042] Optionally, the drone control interface further includes:

[0043] Multiple preset drone formations that can be changed; or,

[0044] Drone formations configured by the user through the user interface.

[0045] Optionally, triggering a preset processing action based on the matching result includes:

[0046] The user interface enters split-screen mode, which is used to display the image captured by the user terminal and the display content corresponding to the processing action.

[0047] Optionally, the method further includes:

[0048] Perform image detection on the drone queue image to determine whether the drone queue image is a real-time captured image;

[0049] Specifically, a preset processing action is triggered only when it is determined that the image of the drone queue is a real-time captured image, based on the matching result.

[0050] Optionally, the method further includes:

[0051] If a match fails, return the result indicating that a match failed.

[0052] According to another aspect of the present invention, an apparatus for processing images of drone platoons is provided, comprising:

[0053] Image acquisition unit, used to acquire images of drone queues captured by user terminals;

[0054] The data processing unit is used to parse the drone queue image and match the parsing results with a preset drone queue database.

[0055] The interaction unit is used to trigger preset processing actions based on the matching results.

[0056] According to another aspect of the present invention, a readable storage medium having executable instructions thereon is provided, which, when executed, cause a computer to perform the above-described method for processing drone queue images.

[0057] According to another aspect of the present invention, a user terminal is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors to process the above-described method for processing drone queue images.

[0058] The technical solution provided by the embodiments of the present invention enables interaction between the terminal and the drone queue, expands the application scenarios of drone performances, and improves the application effect of drones. Attached Figure Description

[0059] The accompanying drawings illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the principles of the invention. These drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification.

[0060] Figure 1 This is a structural block diagram of an exemplary user terminal;

[0061] Figure 2 This is a flowchart illustrating a method for processing drone queue images according to an embodiment of the present invention;

[0062] Figure 3 This is a flowchart illustrating a method for processing drone queue images according to another embodiment of the present invention;

[0063] Figure 4 This is a flowchart illustrating a method for processing drone queue images according to another embodiment of the present invention;

[0064] Figure 5 This is a flowchart illustrating a method for processing drone queue images according to another embodiment of the present invention;

[0065] Figure 6 This is a schematic diagram of a device for processing images of drone queues according to an embodiment of the present invention. Detailed Implementation

[0066] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0067] Figure 1 This is a block diagram of an example user terminal 100 arranged to implement a method for processing images of drone platoons according to the present invention. In a basic configuration 102, the user terminal 100 typically includes a system memory 106 and one or more processors 104. A memory bus 108 can be used for communication between the processors 104 and the system memory 106.

[0068] Depending on the desired configuration, processor 104 can be any type of processor, including but not limited to: microprocessor (μP), microcontroller (μC), digital information processor (DSP), or any combination thereof. Processor 104 may include one or more levels of cache such as L1 cache 110 and L2 cache 112, processor core 114, and registers 116. Example processor core 114 may include an arithmetic logic unit (ALU), floating-point unit (FPU), digital signal processing core (DSP core), or any combination thereof. Example memory controller 118 may be used with processor 104, or in some implementations, memory controller 118 may be an internal part of processor 104.

[0069] Depending on the desired configuration, system memory 106 can be any type of memory, including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 106 may include operating system 120, one or more programs 122, and program data 124. In some embodiments, program 122 may be configured to execute instructions on the operating system using program data 124 by one or more processors 104.

[0070] User terminal 100 may also include an interface bus 140 that facilitates communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to basic configuration 102 via bus / interface controller 130. Example output devices 142 include a graphics processing unit 148 and an audio processing unit 150. They may be configured to facilitate communication with various external devices such as display terminals or speakers via one or more A / V ports 152. Example peripheral interfaces 144 may include a serial interface controller 154 and a parallel interface controller 156, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 158. Example communication devices 146 may include a network controller 160, which may be arranged to facilitate communication with one or more other user terminals 162 via a network communication link through one or more communication ports 164.

[0071] A network communication link can be an example of a communication medium. A communication medium can typically be embodied in a modulated data signal, such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A “modulated data signal” can be a signal whose data set, or whose modifications, can be encoded with information within the signal. As a non-limiting example, a communication medium can include wired media such as wired networks or leased lines, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR), or other wireless media. The term “computer-readable medium” as used herein can include both storage media and communication media.

[0072] The user terminal 100 includes one or more programs 122 that include instructions for executing a method for processing drone queue images according to the present invention.

[0073] Figure 2 An exemplary flowchart illustrates a method for processing drone queue images according to the present invention, the method beginning at step S200.

[0074] First, in step S200, multiple drones take off and arrive at a designated location according to preset commands, displaying preset light colors to form a drone queue image.

[0075] Subsequently, in step S210, images of the drone queue captured by the user terminal are obtained.

[0076] According to an embodiment of the present invention, at a drone performance site, a user can use a user terminal to capture images of the aerial drone formation. The user terminal can be a dedicated terminal (provided by the formation performer), a personal portable terminal such as a mobile phone or tablet, or any personal digital assistant (PDA) device equipped with a camera. To achieve correct response to the drone formation images, the user terminal should pre-install drone formation image processing software, or access a preset online page or online mini-program, and use the corresponding processing software, online page, or online mini-program to call the camera for shooting.

[0077] Subsequently, in step S220, the drone queue image is parsed, and the parsing result is matched against a preset drone queue database. This can be achieved either by the user terminal parsing the drone queue image and matching the result against the preset drone queue database, or by the user terminal transmitting the captured drone queue image to a remote server for parsing and matching the result against the preset drone queue database. According to this embodiment of the invention, the user terminal or the remote server is pre-configured with a drone queue image parsing algorithm. By parsing the drone queue image, the user terminal can find a matching result from the preset drone queue database that matches the currently captured drone queue image.

[0078] Subsequently, in step S230, a preset processing action is triggered based on the matching result.

[0079] According to an embodiment of the present invention, the drone queue library includes not only pre-configured feature information of each drone formation, but also processing action information corresponding to each drone formation after successful matching. Thus, as long as the drone queue performance provider pre-configures each drone formation and the corresponding processing action information after successful matching, the user terminal can scan the drone queue at the drone performance site and perform the processing actions expected by the drone queue performance provider.

[0080] See Figure 3 In one embodiment of the present invention, step S220 includes:

[0081] S310. Analyze the feature point information of the drone queuing image.

[0082] The parsing process can be performed on a user terminal or a remote server. This embodiment of the invention uses a feature point-based method to analyze UAV queue images. Feature points include SIFT, SUTF, or ORB feature points. The parsed image contains multiple feature points, and each UAV image contains one or more feature points. Each feature point has a corresponding descriptor. Through the coordinates of the feature points and / or the descriptor, a mapping relationship can be established between the feature points of the UAV queue image and multiple sets of feature points in a pre-defined UAV queue library.

[0083] S320. Calculate the similarity between the feature point information of the UAV queue image and multiple sets of feature point information in the preset UAV queue database, and determine the set of feature point information with the highest similarity from the multiple sets of feature point information.

[0084] In this embodiment of the invention, feature points of the drone queue image are matched one-to-one with multiple sets of feature points in the drone queue database. Each set of feature points corresponds to a different formation shape. By matching feature points, the similarity between the drone queue image and the corresponding images of each set of feature points in the drone queue database can be effectively determined, thereby achieving matching between the drone image captured on-site and the drone queue database. Since the multiple sets of feature points in the drone queue database correspond to different formation shapes, i.e., the distribution information of each set of feature points is different, the similarity calculation results between the feature points of the drone queue image and the feature points of each set in the drone queue database will differ during the matching process. The result with the highest similarity is taken as the final result.

[0085] Optionally, when calculating similarity, matching feature points are selected based on the Euclidean distance between feature points, and the similarity between the two images is calculated based on the number of matching feature points.

[0086] In this system, the user terminal pre-stores information related to multiple formation shapes in the drone queue library; or, the cloud server stores information related to multiple formation shapes in the drone queue library, and the terminal or remote server retrieves the information stored in the drone queue library from the cloud server after parsing the feature point information of the drone queue image.

[0087] S330. Determine the matching result based on the set of feature points with the highest similarity.

[0088] According to an embodiment of the present invention, the drone queue library is a pre-configured database. The user terminal can determine the ID information corresponding to the drone queue library based on the set of feature points with the highest similarity. Then, it can find other information of the ID in the drone queue library based on the ID information, including processing action information.

[0089] Optionally, the corresponding similarity calculation result is output only when the highest similarity value is higher than a preset threshold; otherwise, a matching failure result is returned. For example, if the highest similarity result is 12%, and the preset threshold is set to 60%, then the current matching can be considered a failure.

[0090] See Figure 4 In one embodiment of the present invention, after step S310, the following is included:

[0091] S410. Determine the target feature points from the feature points of the drone queuing image.

[0092] Target feature points include common feature points shared by various drone queues in the drone queue database. For example, some drones in each queue may be arranged in a specific way, or have specific colors, shapes, etc. The feature points corresponding to this information are marked as target feature points. After parsing the feature points of the drone queue image, the target feature points can be found by using pre-defined rules corresponding to them.

[0093] S420. Determine the transformation matrix of the feature points of the UAV queue image based on the target feature points.

[0094] In this embodiment of the invention, a transformation matrix can be determined by locating target feature points and analyzing the distribution differences of target feature points between the image actually captured by the user terminal and the drone queues in the drone queue database; for example, the transformation matrix can be a homography matrix. The drone queue image is processed using the transformation matrix to map the feature distribution of the image actually captured by the user terminal onto the feature distribution of the drone queues in the drone queue database. This method reduces the influence of environmental factors. For example, if the drone distribution in the user terminal's captured image differs significantly from the original image of the drone queues in the drone queue database due to the shooting angle, this distribution difference can be eliminated through the transformation matrix. Alternatively, when there are deviations in the drone queue formation, the deviation can be adjusted using the transformation matrix to facilitate similarity judgment.

[0095] S430. Transform the feature point information of the UAV queue image according to the transformation matrix.

[0096] The converted drone queue image closely resembles the drone queue situation in the pre-configured drone queue library in terms of both angle and distance, thus significantly improving the matching success rate.

[0097] See Figure 5 In yet another embodiment of the present invention, step S220 includes:

[0098] S510 detects drones in drone queue images using a pre-trained target detection network.

[0099] The target detection network is trained using labeled drone images as training data.

[0100] Optionally, the object detection network is a lightweight neural network that can be loaded and run locally on the user's terminal.

[0101] Optionally, the target detection network runs in the cloud. After the user terminal captures images of the drone queue, it uploads them to the cloud, and the cloud returns the detection results from the target detection network.

[0102] S520. Based on the detection information of the target detection network, determine the coordinate information of each drone in the drone queue image;

[0103] The coordinate information can be the coordinates of the four vertices of the detection frame of the UAV, or the coordinates of the center point of the detection frame. Preferably, the coordinates of the center point of the detection frame are used.

[0104] S530. Match the coordinate information of each drone in the drone queue image with multiple sets of coordinate information in the preset drone queue library.

[0105] Specifically, step S530 includes: matching the coordinate information of each drone in the drone queue image with multiple coordinate ranges included in each set of coordinate information in the preset drone queue library. For example, if the coordinates of a drone a in the drone queue image are (1, 2), and a coordinate range A in a set of coordinate information in the drone queue library is (0, 0) to (2, 2), then it can be determined that drone a is located in coordinate range A, thus confirming that the drone's coordinate point is successfully matched.

[0106] Due to factors such as distance and angle during on-site shooting, the coordinate information of each drone in the drone queue image may differ significantly from the coordinate information of multiple formation shapes stored in the drone queue library. Therefore, before matching, it is necessary to process the coordinate information of each drone in the drone queue image, including scale transformation and angle transformation, to improve the accuracy of matching.

[0107] In one embodiment of the present invention, the length or width of the drone formation can be calculated based on the coordinate information of each drone in the drone formation image. The coordinate information of each drone in the drone formation image can be adjusted according to the ratio of the length or width of the drone formation to the length or width of multiple formation shapes stored in the drone formation library, thereby improving the accuracy of matching.

[0108] For example, the system can acquire the sensing data and positioning data from the three-axis accelerometer when the user terminal captures images of the drone queue, as well as the positioning data of the drone queue. This allows the system to determine the user terminal's shooting orientation relative to the drone queue, including information on the shooting angle and position. Based on the user terminal's shooting orientation, the coordinate information of each drone in the drone queue image can be mapped to a coordinate system consistent with the drone queue database, thereby completing scale and angle transformations and improving matching accuracy. Specifically, the above transformations can be performed based on a homography matrix.

[0109] Regarding steps S210 and S220, since drone formation performances achieve pattern changes through the positions and light colors of multiple drones in the air, the drone formation images obtained by different users at different times or from different angles at the same time will vary. Furthermore, given the special nature of drone formation performances, there are generally specific formation performance records within a certain geographical area. Therefore, when a user's terminal enters that geographical area, it can access a dedicated network. When the user takes a picture, the network automatically connects to obtain the preset shape of the formation at the current moment and matches it with the analyzed and identified captured image. If they match, the image is searched in the database; if they do not match, the image is searched in the database according to the preset image obtained from the network at the current moment.

[0110] According to an embodiment of the present invention, after a drone queue image successfully matches a preset drone queue database, the user terminal can trigger any processing action, including: loading a preset link in the user interface; or loading a preset game in the user interface; or loading a red envelope animation in the user interface; or loading a product animation in the user interface; or displaying a preset pattern in the user interface; or displaying preset text in the user interface; or displaying a drone control interface in the user interface; wherein the drone control interface includes interactive buttons for drone formation changes. The processing actions triggered by the user terminal can be preset in the user terminal program or obtained by linking to a third-party server.

[0111] For example, each drone formation corresponds to a different product link. After successfully parsing the drone formation image, the user terminal is redirected to the corresponding product page or a gift page, including discount coupons, cash coupons, and vouchers. Another example is that after successfully parsing the drone formation image, a red envelope animation can be displayed, which the user can click to claim. Yet another example is that after successfully parsing the drone formation image, the user terminal provides a drone control interface, allowing the user to control the drones to change formations. For yet another example, if the drone formation resembles a piano, the user terminal displays a piano-playing game; if the formation resembles a snake, a snake-playing game is displayed.

[0112] In another embodiment of the present invention, after the drone queue image is successfully matched with a preset drone queue library, the user terminal can enter a split-screen mode to display the image captured by the camera and the display content corresponding to the processing actions. Specifically, half of the user terminal's display screen can be used to display the real-time drone performance, and the other half can be used to display product links, videos, games, and other interfaces.

[0113] According to embodiments of the present invention, when the user terminal provides a drone control interface, the drone control interface displays interactive buttons for drone formation changing, as well as multiple preset drone formations that can be changed, thus facilitating user control of the drone formation. Alternatively, when the user terminal provides a drone control interface, the drone control interface displays interactive buttons for drone formation changing, as well as a drone formation configuration page, so that the user can define drone formations in real time on the user terminal and control the drones to change formations according to the user's expectations.

[0114] Specifically, after the user terminal successfully matches the drone queue image with the preset drone queue library, it can display the real-time information of the currently captured drone queue on the terminal interface. At the same time, it can split the screen, and the other half of the screen can display other graphic information. Users can select the target graphic and the corresponding aerial drone formation by sliding on the side or in the split screen interface to perform graphic transformation.

[0115] According to an embodiment of the present invention, the method further includes: a user terminal performing image detection on the drone queue to determine whether the drone queue image is a real-time captured image; wherein, only when it is determined that the drone queue image is a real-time captured image, the user terminal triggers a preset processing action based on the matching result. Specifically, it can be determined whether the user is capturing a real scene or a screen by detecting stripes or pixels in the image.

[0116] Furthermore, in another embodiment of the present invention, after a matching failure, the result of the matching failure is returned.

[0117] The technical solutions provided by the embodiments of the present invention can enhance the interaction between users and drones and increase user stickiness.

[0118] See Figure 6 This invention provides a processing apparatus for drone queue images, comprising:

[0119] Image acquisition unit 610 is used to acquire images of drone queues captured by the user terminal;

[0120] The data processing unit 620 is used to parse the drone queue image and match the parsing results with a preset drone queue library;

[0121] The interaction unit 630 is used to trigger a preset processing action based on the matching result.

[0122] Optionally, when the data processing unit 620 parses the drone queue image and matches the parsing result with a preset drone queue database, it specifically performs the following:

[0123] Analyze the feature point information of the drone queue image;

[0124] Calculate the similarity between the feature point information of the drone queue image and multiple sets of feature point information in a preset drone queue library, and determine the set of feature point information with the highest similarity from the multiple sets of feature point information;

[0125] The matching result is determined based on the set of feature points with the highest similarity.

[0126] Optionally, after parsing the feature point information of the UAV queuing image, the data processing unit 620 is further configured to:

[0127] Target feature points are determined from the feature points of the drone queuing image;

[0128] Determine the transformation matrix of the feature points of the UAV queue image based on the target feature points;

[0129] The feature point information of the drone queue image is transformed according to the transformation matrix.

[0130] Optionally, when the data processing unit 620 parses the drone queue image and matches the parsing result with a preset drone queue database, it specifically performs the following:

[0131] The drones in the drone queue image are detected by a pre-trained target detection network.

[0132] Based on the detection information from the target detection network, the coordinate information of each drone in the drone queue image is determined;

[0133] The coordinate information of each drone in the drone queue image is matched with multiple sets of coordinate information in a preset drone queue library.

[0134] Optionally, when the data processing unit 620 matches the coordinate information of each drone in the drone queue image with multiple sets of coordinate information in a preset drone queue database, it specifically performs the following:

[0135] The coordinate information of each drone in the drone queue image is matched with multiple coordinate ranges included in each set of coordinate information in the preset drone queue library.

[0136] Optionally, before matching the coordinate information of each drone in the drone queue image with multiple sets of coordinate information in a preset drone queue database, the data processing unit 620 is further configured to:

[0137] Calculate the length or width of the drone queue based on the coordinate information of each drone in the drone queue image;

[0138] Based on the length or width of the drone queue and the length or width corresponding to multiple sets of coordinate information in the preset drone queue library, determine the adjustment ratio of the coordinate information of the drone queue.

[0139] Adjust the coordinate information of each drone in the drone queue image according to the adjustment ratio of the drone queue coordinate information.

[0140] Optionally, before matching the coordinate information of each drone in the drone queue image with multiple sets of coordinate information in a preset drone queue database, the data processing unit 620 is further configured to:

[0141] Acquire the positioning data of the drone queue, the sensing data of the three-axis accelerometer of the user terminal, and the positioning data of the user terminal;

[0142] Based on the positioning data of the drone queue, the sensing data of the three-axis accelerometer of the user terminal, and the positioning data of the user terminal, the shooting position of the user terminal relative to the drone queue is determined.

[0143] The coordinate information of each drone in the drone queue image is adjusted according to the shooting position of the user terminal relative to the drone queue.

[0144] Optionally, the interaction unit 630 is specifically used for:

[0145] After a successful match, obtain the matching information from the drone queue library;

[0146] Based on the matching information, a preset processing action is triggered;

[0147] The preset processing actions include:

[0148] Load the preset links in the user interface; or,

[0149] Load a preset game into the user interface; or,

[0150] Load the red envelope animation in the user interface; or,

[0151] Load product animations in the user interface; or,

[0152] Display a preset pattern on the user interface; or,

[0153] Display preset text in the user interface; or,

[0154] The user interface displays the drone control interface; wherein the drone control interface includes interactive buttons for drone formation changes.

[0155] Optionally, the drone control interface further includes:

[0156] Multiple preset drone formations that can be changed; or,

[0157] Drone formations configured by the user through the user interface.

[0158] Optionally, the interaction unit 630 is specifically used for:

[0159] Enter split-screen mode, which is used to display the image captured by the camera and the display content corresponding to the processing action.

[0160] Optionally, the data processing unit 620 is also used for:

[0161] Perform image detection on the drone queue image to determine whether the drone queue image is a real-time captured image;

[0162] Specifically, the interaction unit 630 is triggered only when it is determined that the image of the drone queue is a real-time captured image.

[0163] Optionally, the data processing unit 620 is also used for:

[0164] If a match fails, return the result indicating that a match failed.

[0165] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embedded in a tangible medium, such as a floppy disk, CD-ROM, hard disk, or any other machine-readable storage medium, wherein when the program is loaded into and executed by a machine such as a computer, that machine becomes an apparatus for practicing the present invention.

[0166] When the program code is executed on a programmable computer, the user terminal generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute various methods of the present invention according to instructions in the program code stored in the memory.

[0167] By way of example, and not limitation, computer-readable media include computer storage media and communication media. Computer storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Any combination of the above is also included within the scope of computer-readable media.

[0168] It should be understood that, in order to simplify the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this method of the invention should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0169] Those skilled in the art will understand that modules, units, or components of the devices in the examples of this invention can be arranged in the devices described in this embodiment, or alternatively, can be located in one or more devices different from the devices in this example. The modules in the foregoing examples can be combined into a single module or, in addition, can be divided into multiple sub-modules.

[0170] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features of the invention in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device of such invention. Unless expressly stated otherwise, each feature of the invention in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0171] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention and form different embodiments.

[0172] Furthermore, some of the embodiments described herein are methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing the functions. Therefore, a processor having the necessary instructions for implementing the methods or method elements forms means for implementing the methods or method elements. Furthermore, the elements described herein in the apparatus embodiments are examples of means for implementing the functions performed by elements for the purposes of carrying out the invention.

[0173] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

[0174] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and edibility purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The invention described herein is illustrative and not restrictive, and its scope is defined by the appended claims.

Claims

1. A method for processing images of drone queues, characterized in that, include: Acquire images of drone queues captured by the user terminal; The process involves parsing the drone queue image and matching the parsing results with a pre-defined drone queue database, including: Determine the coordinate information of each drone in the drone queue image; Based on the positioning data of the drone queue, the sensing data of the three-axis accelerometer of the user terminal, and the positioning data of the user terminal, the shooting position of the user terminal relative to the drone queue is determined, and the coordinate information of each drone in the drone queue image is adjusted according to the shooting position. Based on the adjusted coordinate information of each drone, calculate the length or width of the drone queue; determine the adjustment ratio based on the ratio of the length or width to the corresponding length or width in the preset drone queue library; and adjust the coordinate information of each drone proportionally according to the adjustment ratio. The coordinate information of each drone after the ratio adjustment is matched with multiple coordinate ranges included in each set of coordinate information in the preset drone queue library to determine the matching result; Determine whether the image of the drone queue is a real-time image taken by the audience at the scene; If so, then trigger the preset processing action based on the matching result; The processing action includes at least: displaying a drone control interface on the user interface; wherein the drone control interface includes: interactive buttons for the audience to control drone formation changes; and, a plurality of preset drone formations for the audience to select for drone formation changes; or, drone formations configured by the audience on the user interface.

2. The method as described in claim 1, characterized in that, The step of parsing the drone queue image and matching the parsing results with a preset drone queue database includes: Analyze the feature point information of the drone queuing image; Calculate the similarity between the feature point information of the drone queue image and multiple sets of feature point information in a preset drone queue library, and determine the set of feature point information with the highest similarity from the multiple sets of feature point information; The matching result is determined based on the set of feature points with the highest similarity. The highest value of the similarity is higher than a preset threshold.

3. The method as described in claim 2, characterized in that, After parsing the feature point information of the drone queue image, the method further includes: determining a preset target feature point from the feature points of the drone queue image; The transformation matrix of the feature points of the UAV queue image is determined based on the target feature points; the feature point information of the UAV queue image is transformed based on the transformation matrix.

4. The method as described in claim 1, characterized in that, Determining the coordinate information of each drone in the drone queue image includes: The drones in the drone queue image are detected by a pre-trained target detection network. Based on the detection information from the target detection network, the coordinate information of each drone in the drone queue image is determined.

5. The method according to any one of claims 1-4, characterized in that, The preset processing actions include: Load the preset links in the user interface; or, Load a preset game into the user interface; or, Load the red envelope animation in the user interface; or, Load product animations in the user interface; or, Display a preset pattern on the user interface; or, Display preset text in the user interface.

6. The method as described in claim 5, characterized in that, The step of triggering a preset processing action based on the matching result includes: The user interface enters split-screen mode, which is used to display the image captured by the user terminal and the display content corresponding to the processing action.

7. The method as described in claim 1, characterized in that, Also includes: If a match fails, return the result indicating that a match failed.

8. A processing apparatus for drone queue images, characterized in that, include: Image acquisition unit, used to acquire images of drone queues captured by user terminals; The data processing unit is used to parse the drone queue image and match the parsing results with a preset drone queue database, including: Determine the coordinate information of each drone in the drone queue image; Based on the positioning data of the drone queue, the sensing data of the three-axis accelerometer of the user terminal, and the positioning data of the user terminal, the shooting position of the user terminal relative to the drone queue is determined, and the coordinate information of each drone in the drone queue image is adjusted according to the shooting position. Based on the adjusted coordinate information of each drone, calculate the length or width of the drone queue; determine the adjustment ratio based on the ratio of the length or width to the corresponding length or width in the preset drone queue library; and adjust the coordinate information of each drone proportionally according to the adjustment ratio. The coordinate information of each drone after the ratio adjustment is matched with multiple coordinate ranges included in each set of coordinate information in the preset drone queue library to determine the matching result; The determining unit is used to determine whether the image of the drone queue is an image taken in real time by the audience at the scene; An interactive unit is used to trigger a preset processing action based on the matching result if the match is found to be true. The processing action includes at least: displaying a drone control interface on the user interface; wherein the drone control interface includes: interactive buttons for the audience to control drone formation changes; and, a plurality of preset drone formations for the audience to select for drone formation changes; or, drone formations configured by the audience on the user interface.

9. A readable storage medium, characterized in that, It has executable instructions that, when executed, cause a computer to perform the method as claimed in any one of claims 1-7.

10. A user terminal, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors as included in any one of claims 1-7.

Images