Director video optimization method and device, storage medium and image acquisition equipment

Abstract

The invention discloses a director video optimization method and device, a storage medium and image acquisition equipment. The method comprises the following steps: determining an original film source of a current director film; inputting the original fragment into an optimized event tracking model to obtain an optimized event code output by the optimized event tracking model; the optimized event tracking model is obtained by training according to a prediction event code and a standard event code of the original film source, the standard event code is used for identifying a standard key event in the original film source and triggering time of the standard key event, and the prediction event code is a prediction result output by the initial event tracking model to the original film source; and according to the optimized event code, generating an optimized director fragment for the original fragment source. According to the embodiment of the invention, the existing problems can be effectively repaired.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, storage medium and image acquisition device for optimizing a broadcast video. Background Art

[0002] In the intelligent education scenario, in order to fully record and present the teaching process with high quality, the intelligent classroom is equipped with a multi-camera shooting system, a smart blackboard and a director host. Each camera can shoot panoramic and close-up images of different characters and objects at different angles; the smart blackboard is used to intelligently demonstrate the details of the blackboard; the director host splices the film sources in real time, and realizes the intelligent recording of the teaching process by flexibly switching between panoramic and close-up images and synthesizing the director video.

[0003] It can be seen that the quality of the director video can directly affect the recording effect of the teaching scene and content. Therefore, a director video optimization method is needed to synthesize high-quality director videos. Summary of the invention

[0004] The present application provides a director video optimization method, device, storage medium and image acquisition equipment, which can solve the technical problem of poor director video quality in related technologies.

[0005] In the first aspect, an embodiment of the present application provides a director video optimization method, which is applied to an image acquisition device, and the method includes: determining the original film source corresponding to the current director's film; inputting the above-mentioned original clip into an optimization event tracking model to obtain an optimization event code output by the above-mentioned optimization event tracking model; the above-mentioned optimization event tracking model is obtained by training the initial event tracking model according to the predicted event code and standard event code of the above-mentioned original film source, and the above-mentioned standard event code is used to identify the standard key events in the above-mentioned original film source and the trigger time of the above-mentioned standard key events, and the above-mentioned predicted event code is the prediction result of the above-mentioned initial event tracking model output for the above-mentioned original film source; sending the above-mentioned optimization event code to the director device, and the above-mentioned optimization event code is used to generate an optimized director's film corresponding to the above-mentioned original film source.

[0006] The beneficial effects brought about by the technical solution of the first aspect above include at least: adding correct and accurate standard event codes to the source of the director's film that performs poorly in the actual scene to identify the key events and trigger times in the source; then training the model multiple times through real sources with standard labels, so that the converged model can accurately identify various key events in the original source. Further, based on the optimized model, the event code is re-output for the original source, so that the director device generates an optimized director's film according to the optimized event code, thereby effectively repairing the current director's film that performs poorly and solving the problem of poor current film effects.

[0007] In some possible implementations, the method further includes: extracting a close-up picture of the target subject in the original film source according to the optimization event code, and sending the close-up picture to the director device, where the close-up picture is used to generate an optimized director film corresponding to the original film source.

[0008] Through the technical solutions in the above possible implementations, when regenerating the current director's finished film, the close-up images of the target shooting object can be pre-cut out from the original film source according to the optimized event code, and the parts with more important information are retained while the redundant parts in the original film source are removed. The close-up images are then sent to the director's device, so that it can regenerate the director's finished film through more accurate event codes and pre-cut close-up images, and replace the original problematic director's finished film with the repaired director's finished film, so that users can use the optimized director's finished film and improve the user experience.

[0009] In some possible implementations, the method further includes: obtaining a standard event code of the original film source; using the original film source as a sample in an image acquisition device simulation environment to train the initial event tracking model; and obtaining the optimized event tracking model when a preset matching condition is satisfied between the predicted event code and the standard event code.

[0010] Through the technical solutions in the above possible implementation methods, the original film source with correct and accurate standard event codes is used as a sample, and the initial event tracking model is trained for multiple iterations, so that the training and parameter adjustment direction of the model can be closely integrated with the problem scene. When the predicted event code output by the initial event tracking model for the original film source meets the preset matching conditions with the standard event code, it means that the model can accurately identify the key events in the original film source, that is, the optimized event tracking model is obtained through the optimization link of the tracking director algorithm driven by real data.

[0011] In some possible implementations, the above-mentioned obtaining of the standard event code of the above-mentioned original film source includes: determining the key segment material corresponding to at least one abnormal scene in the above-mentioned currently directed film in the above-mentioned original film source, and obtaining the standard event code corresponding to each key segment material; the above-mentioned using the above-mentioned original film source as a sample in the image acquisition device simulation environment to train the above-mentioned initial event tracking model includes: using the above-mentioned each key segment material as a sample in the image acquisition device simulation environment to train the above-mentioned initial event tracking model.

[0012] Through the technical solutions in the above possible implementation methods, when training the initial event tracking model, considering that there are some clips in the huge original film source in which no key events have occurred or have been correctly identified and edited, then these clips are not very helpful in algorithm optimization. Therefore, the key clip materials corresponding to the abnormal scenes in the current directed film can be cut out in advance, and standard event codes can be added to the screened key clip materials in a targeted manner. Then, these key clip materials are used as samples for subsequent model training, so as to improve the efficiency of model training.

[0013] In some possible implementations, the above-mentioned original film source is used as a sample in the image acquisition device simulation environment to train the above-mentioned initial event tracking model, including: in the image acquisition device simulation environment, the above-mentioned original film source is stored in the input data stream file of the above-mentioned initial event tracking model; determining the predicted event code output by the above-mentioned initial event tracking model for the above-mentioned original film source, and judging whether the above-mentioned predicted event code and the above-mentioned standard event code meet the preset matching conditions; if not, adjusting the parameters of the above-mentioned initial event tracking model based on the above-mentioned predicted event code and the above-mentioned standard event code.

[0014] Through the technical solutions in the above possible implementation methods, when the initial event tracking model in the simulation environment starts running, its input data comes from the real-time collected data of the simulation device. In order to allow the model to receive sample data as input, it is necessary to replace the original film source with the input data stream file of the initial event tracking model, so that the initial event tracking model can output the predicted event code for the original film source with the original film source as input data. When the predicted event code output by the model and the standard event code still meet the preset matching conditions, it means that the model performance has not yet been fitted to expectations. At this time, the initial event tracking model can be adjusted according to the predicted event code and the standard event code to optimize the various network parameters in the model.

[0015] In some possible implementations, the method further includes: acquiring the optimized event tracking model; and replacing the initial event tracking model with the optimized event tracking model.

[0016] Through the technical solutions in the above possible implementation methods, in addition to directly training the model on the image acquisition device, the model can also be trained and converged on other devices that support the model; after the model converges, the image acquisition device obtains the optimized model and replaces the originally deployed initial model with the optimized model, so that the image acquisition device can output event codes for the film sources collected later based on the optimized event tracking model, thereby continuously improving the quality of subsequent directed films.

[0017] In some possible implementations, the above-mentioned determining the original film source corresponding to the currently directed film includes: determining that the film source corresponding to the above-mentioned image acquisition device among all the film sources corresponding to the currently directed film is the above-mentioned original film source.

[0018] Through the technical solutions in the above possible implementation methods, in actual application scenarios, the director's film can include pictures captured by multiple image acquisition devices on multiple different shooting objects. Then for a single device, when determining the original film source, you can first determine all the film sources used for the current director's film. Among these film sources, the film source with the local image acquisition device identifier is the original film source collected by this device for the current director's film. Then this device should use the original film source it has collected to optimize the initial event tracking model deployed by itself in a targeted manner.

[0019] In a second aspect, an embodiment of the present application provides a video director optimization device, which is applied to an image acquisition device, and the device includes:

[0020] A data acquisition module is used to determine the original film source corresponding to the current directed film; a model output module is used to input the above original clip into the optimized event tracking model to obtain the optimized event code output by the above optimized event tracking model; the above optimized event tracking model is obtained by training the initial event tracking model according to the predicted event code and standard event code of the above original film source, the above standard event code is used to identify the standard key events in the above original film source and the triggering time of the above standard key events, and the above predicted event code is the prediction result of the above initial event tracking model output for the above original film source; a film repair module is used to send the above optimized event code to the directed film device, and the above optimized event code is used to generate an optimized directed film corresponding to the above original film source.

[0021] In a third aspect, an embodiment of the present application provides an image acquisition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0022] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on an image acquisition device, the image acquisition device executes the method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 An exemplary system architecture diagram of a method for optimizing a broadcast video provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for optimizing a broadcast video provided in an embodiment of the present application;

[0026] Figure 3 A logical framework diagram of a director video repair process provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a flow chart of model training in a broadcast video optimization method provided in an embodiment of the present application;

[0028] Figure 5 A logic flow chart of simulation training steps of an event tracking model provided in an embodiment of the present application;

[0029] Figure 6 A structural block diagram of a video director optimization device provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the structure of an image acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0032] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. And in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0033] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0034] In the increasingly intelligent education scene, in order to ensure the comprehensive recording and high-quality presentation of the teaching process, a shooting system consisting of multiple high-performance cameras that can shoot various images of different people and angles in the classroom is usually deployed in the intelligent classroom, as well as a smart blackboard that makes the teaching blackboard demonstration more intelligent and convenient. In addition, it also includes a director host that can synthesize high-quality recorded videos based on multiple different sources.

[0035] Each camera in the shooting system has its own function. The student camera is used to capture the panoramic picture of the students, and to capture the close-up picture of the students from the panoramic picture when key events occur (such as raising hands, answering questions, etc.). The teacher camera focuses on the teacher, and is used to capture the panoramic picture of the teacher, and to capture the close-up picture of the teacher from the panoramic picture when key events occur (such as writing on the blackboard, explaining questions, etc.).

[0036] The director host splices the rich video sources captured in real time based on the current scene, and uses advanced director technology to synthesize the panoramic picture and close-up picture to flexibly switch between the director's pictures. The director's pictures in the film fully and vividly show the teaching scene and teaching content, thus realizing efficient and accurate recording of the on-site teaching process.

[0037] Generally speaking, in scenarios where a composite directing film is required, the image acquisition device (such as the student camera and teacher camera mentioned above) will analyze and identify the video data it has collected, and add corresponding event codes when key events occur to identify the event type and the event that occurred. These event codes and video data will be output together to the directing host. The directing host cuts and splices multiple video data based on the event code to obtain a directing film.

[0038] However, the synthesis of the director's film is still facing challenges and problems that cannot be ignored. When users report that the director's film is poor, the director's screen is often optimized through network sources or custom materials, but these elements are not collected on-site, which makes users feel that the director's film cannot reflect the real situation. In addition, in the face of flaws in the director's screen, it currently mainly relies on the technical staff's post-manual adjustment and optimization. Although this method can make up for the technical deficiencies to a certain extent, the high labor costs, time-consuming processing procedures, and the fact that only one director's screen can be customized at a time have limited the widespread application of this optimization method in large-scale, high-frequency educational recording and broadcasting scenarios.

[0039] In view of this, the embodiment of the present application provides a method for optimizing director video, which trains the model iteratively multiple times through real film sources with standard labels, so that the converged model can accurately identify various key events in the original film source. Then, based on the optimized model, the event code is re-output for the original film source, so that the director device generates an optimized director film according to the optimized event code, thereby effectively repairing the current director film with poor performance, so as to solve the technical problem of poor director video quality.

[0040] See also Figure 1 , Figure 1 A system architecture diagram of a system applicable to a director video optimization method provided in an embodiment of the present application.

[0041] like Figure 1 As shown, the system includes an image acquisition device 101, a network 102, and a broadcast director device 103. The functions in the image acquisition device 101 can be implemented by hardware or by software. The image acquisition device 101 can be connected to the broadcast director device 103 through the network 102. The network 102 is used to provide a medium for a communication link between the image acquisition device 101 and the broadcast director device 103. The network 102 may include various types of wired communication links or wireless communication links, for example: the wired communication link includes an optical fiber, a twisted pair or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link or a microwave communication link, etc.

[0042] The image acquisition device 101 can receive messages from the broadcast director device 103 through the network 102, or send messages to the broadcast director device 103, or the image acquisition device 101 can interact with the broadcast director device 103 through the network 102, and then receive messages or data sent by other image acquisition devices to the broadcast director device 103. The functions in the broadcast director device 103 can be implemented by hardware or software.

[0043] In an embodiment of the present application, when a directed film of poor quality appears in an application scenario, the image acquisition device 101 first determines the original source of the picture of the current directed film; next, the image acquisition device 101 inputs the original clip into the optimized event tracking model to obtain the optimized event code output by the optimized event tracking model; the optimized event tracking model is obtained by training the initial event tracking model according to the predicted event code and the standard event code of the original film source, the standard event code is used to identify the standard key events and the triggering time of the standard key events in the original film source, and the predicted event code is the prediction result of the initial event tracking model for the original film source output; finally, the image acquisition device 101 sends the optimized event code to the directed film device 103 through the network 102, and the directed film device 103 can generate an optimized directed film corresponding to the original film source according to the optimized event code.

[0044] It should be understood that Figure 1 The number of image acquisition devices 101, networks 102 and broadcast director devices 103 is only illustrative, and any number of image acquisition devices, networks and broadcast director devices may be used according to implementation requirements.

[0045] Combination Figure 1 , see Figure 2 , Figure 2 A flowchart of a director video optimization method provided for an embodiment of the present application. The execution subject of the embodiment of the present application can be an image acquisition device that executes the director video optimization, or a processor in the image acquisition device that executes the director video optimization method, or a director video optimization service in the image acquisition device that executes the director video optimization method. For the convenience of description, the specific execution process of the director video optimization method is introduced below by taking the execution subject as an image acquisition device as an example.

[0046] like Figure 2 As shown, the broadcast video optimization method may at least include:

[0047] S202: The image acquisition device determines the original film source corresponding to the current directed film.

[0048] In the embodiment of the present application, in order to optimize the current directed film, it is first necessary to determine the original film source to which the current directed film belongs. When there is only a single image acquisition device in the recording scene to capture the screen of the directed film, it can be determined that the film source of the current directed film is the original film source of the image acquisition device.

[0049] In some possible embodiments, when the recording scene includes multiple image acquisition devices, the directed film will also include multiple images captured by multiple image acquisition devices, that is, the current directed film is obtained from the film sources collected by multiple image acquisition devices. And because the images of each image acquisition device are different, the event tracking models therein will also be different, so each image acquisition device should use the original film source it has collected to optimize its own initial event tracking model in a targeted manner.

[0050] Furthermore, among the multiple image acquisition devices corresponding to the current directed film, the image acquisition device currently executing the director video optimization method (any one of the multiple image acquisition devices) also needs to determine the film source belonging to this image acquisition device (that is, the film source acquired by the image acquisition device itself) as the original film source from all the film sources corresponding to the current directed film.

[0051] For example, the image acquisition device is a camera. When the recording scene includes a student camera and a teacher camera, the student film source captured by the student camera is used to train the model in the student camera, and the teacher film source captured by the teacher camera is used to train the model in the teacher camera.

[0052] Specifically, each image acquisition device sends its own image acquisition device ID to the director device, so that the director device can clearly know which device the source file comes from based on the image acquisition device ID of the source file, and then reasonably edit the images of different objects according to its own film logic to obtain the director's film. Then for the current image acquisition device, it can first determine all the image acquisition device sources corresponding to the current director's film and the image acquisition device ID of each image acquisition device source, among which the image acquisition device source with the local image acquisition device ID is the original source required for its own model training.

[0053] S204, the image acquisition device inputs the original clip into the optimized event tracking model to obtain the optimized event code output by the optimized event tracking model; the optimized event tracking model is obtained by training the initial event tracking model according to the predicted event code and standard event code of the original film source, the standard event code is used to identify the standard key events and the triggering time of the standard key events in the original film source, and the predicted event code is the prediction result of the initial event tracking model output for the original film source.

[0054] Optionally, the image acquisition device can use the original film source of the problem director to optimize and update the existing initial event tracking model, so that the training and parameter adjustment direction of the model can be closely combined with the actual situation of the problem site, and the optimization link of the tracking and directing algorithm driven by real data can enable the event tracking model to more effectively solve the actual problems on site. In the embodiment of the present application, the optimized event tracking model can be obtained by training the initial event tracking model according to the predicted event code and standard event code of the original film source. As the name implies, the function of the standard event code is to identify the standard key events and the triggering time of the standard key events in the original film source, that is, it can reflect the actual situation in the original film source. The predicted event code is the prediction result of the initial event tracking model output to the original film source. After the standard event code label and the predicted event code label, the initial event tracking model can be effectively adjusted, so that the model fits to the expected performance to obtain the optimized event tracking model. Further, the original film source is input into the optimized event tracking model, and the optimized event code that is more accurate and more in line with the actual situation can be obtained through the optimized model.

[0055] In some possible embodiments, the training process can be directly executed on the image acquisition device, or the model can be trained and converged on other devices that support the deployment of event tracking models. In the scenario where other devices execute, after obtaining the optimized event tracking model, the other devices send the optimized event tracking model to the image acquisition device. After the image acquisition device obtains the optimized event tracking model, it replaces the initial event tracking model with the optimized event tracking model, and outputs event codes for the film sources collected later based on the optimized event tracking model, thereby continuously improving the quality of subsequent directed films.

[0056] S206. The image acquisition device sends an optimization event code to the broadcasting device. The optimization event code is used to generate an optimized broadcasting film corresponding to the original film source.

[0057] Optionally, when optimizing the current problematic directing film, the image acquisition device needs to send the optimization event code regenerated based on the optimization event tracking model to the directing device, so that the directing device can regenerate the optimized directing film corresponding to the original film source based on the optimization event code.

[0058] In some possible embodiments, if the subject of the image acquisition device is a specific person, then as some events occur, it may be necessary to present close-up images of these subjects in the director's film. Therefore, in order to facilitate the director to directly use the close-up images related to the event, please refer to Figure 3In the logical framework diagram of a director video repair process provided in an embodiment of the present application, the image acquisition device can further accurately capture close-up pictures from the original film source according to the optimized event code after obtaining the optimized event code, and in addition to sending the optimized event code to the director device, the close-up picture is also transmitted to the director device, so that the director device can select the appropriate picture to re-synthesize the optimized director film according to the close-up picture and the optimized event code received from the image acquisition device and the director film logic in its own director module, and replace the original current director film with the optimized director film.

[0059] In another feasible implementation, if the broadcast director device itself has simulation capabilities, the optimized model can also be deployed to the simulation module of the broadcast director device, so that the broadcast director device can use its own idle computing power to regenerate the optimized event code and close-up picture, and then regenerate the broadcast film with more accurate event codes and close-up pictures, and then perform a replacement operation to seamlessly replace the current broadcast film with a new repaired version, solving the current problem faced by the user. The embodiment of the present application does not make a unique limitation on the hardware device carrier that regenerates the optimized event code and close-up picture in the original film source during the repair process.

[0060] In summary, after the image acquisition device determines the original source corresponding to the current director's finished film, it inputs the original clip into the optimized event tracking model to obtain the optimized event code output by the optimized event tracking model; the optimized event tracking model is obtained by training the predicted event code and standard event code of the original source using the initial event tracking model, the standard event code is used to identify the standard key events and the triggering time of the standard key events in the original source, and the predicted event code is the prediction result of the initial event tracking model output on the original source; the optimized event code is sent to the director's device, and the optimized event code is used to generate the optimized director's finished film corresponding to the original source. Add correct and accurate standard event codes to the sources of the director's finished films that perform poorly in actual scenes to identify the key events and triggering times in the source; then the model is iteratively trained multiple times through real sources with standard labels, so that the converged model can accurately identify various key events in the original source. Further, based on the optimized model, the event code is re-output for the original film source, so that the director equipment generates an optimized director film according to the optimized event code, thereby effectively repairing the current director film with poor performance and solving the problem of poor current film effect.

[0061] In the embodiment of the present application, the training process of the initial event tracking model can be performed directly on the image acquisition device, or the model can be trained and converged on other devices that support the deployment of the event tracking model. The following takes the model training on the image acquisition device as an example to describe the model training process in detail. Figure 4 , Figure 4A flowchart of model training in a director video optimization method provided in an embodiment of the present application.

[0062] like Figure 4 As shown, the model training step in the director video optimization method may at least include:

[0063] S402: The image acquisition device obtains the standard event code of the original film source.

[0064] Optionally, when the original film source is used for model training, the image acquisition device first obtains the standard event code that identifies the standard key events and event triggering time in the original film source. Taking the classroom teaching scene as an example, the standard key events should be student answer events, free discussion events, teacher blackboard events, etc. that appear in the classroom, which are determined according to the actual scene requirements. The standard event code identifies the type information and time information of these events. The standard event code can be manually screened from the original film source, and the manual marking method is conducive to ensuring the correctness of the standard event code. Of course, some tools can also be used for automatic marking, and the specific situation of the tool is not limited in the embodiment of the present application. These standard event codes are used as standard labels for sample data. The output results of the model will be fitted toward the standard event code during the continuous iteration process, and finally when the model converges, more accurate event codes can be output to the real film source, thereby solving the problems faced by the synthetic director in the actual scene.

[0065] Specifically, see Figure 5 , Figure 5 The following is a logic flow chart of the simulation training steps of an event tracking model provided in an embodiment of the present application. Figure 5 As shown, the image acquisition device first retrieves the original source of the current director's finished film, and then marks the key events that occur therein according to the original source, thereby obtaining a standard event code for guiding the direction of model training. However, since the original source is a long-term continuous video collected by the image acquisition device in the recording scene, considering that there are some clips in the huge original source in which no key events have occurred or have been correctly identified and edited, these clips are not very helpful in algorithm optimization. Therefore, the key clip materials corresponding to the abnormal pictures in the current director's finished film can be intercepted in advance, and standard event codes can be added to the selected key clip materials in a targeted manner. Then, these key clip materials are used as samples for subsequent model training to improve the efficiency of model training. It should be noted that when intercepting key clip materials, accurate time code positioning technology can be used to ensure that the start and end points of the clips can accurately contain the start and end of the event when intercepting, thereby ensuring the integrity of key information.

[0066] S404: The image acquisition device uses the original film source as a sample in the image acquisition device simulation environment to train an initial event tracking model.

[0067] Furthermore, a more realistic training environment is conducive to the convergence of the model, so an image acquisition device simulation environment can be built in advance to implement the training of the initial event tracking model in the image acquisition device simulation environment. The simulation environment can simulate the entire process from video capture to event recognition and tracking, aiming to allow the model to adjust parameters and iterate in a more realistic environment. The simulation training method is more conducive to improving the model's ability to understand the video content and the response speed, and ultimately achieve efficient and accurate event tracking. During the training process, the model in the image acquisition device will continuously adjust its internal parameters and algorithm logic to more accurately predict and identify each standard key event and its triggering moment in the video.

[0068] Optionally, in the image acquisition device simulation environment, the image acquisition device starts running and uses the simulated "lens" as a collector of model input data. In order for the model to receive sample data as input, the original film source needs to be stored in the input data stream file of the initial event tracking model, so that the initial event tracking model can use the original film source as input data to output predicted event codes for the original film source.

[0069] In a preferred embodiment, see Figure 5 , multiple key segment materials including standard key events have been intercepted from the original film source and standard event codes have been added to each key segment material. Then, the key segment materials can be directly replaced into the data stream continuously collected by the simulation image acquisition device. The initial event tracking model can then output the predicted event code for the key segment materials without the need to identify and analyze the redundant content in the original film source.

[0070] For further information, see Figure 5 , the image acquisition device uses the key clip material as the input of the initial event tracking model, thereby obtaining the predicted event code output by the initial event tracking model for the key clip material. The predicted event code output by the event tracking model represents the model's recognition and understanding of the key events that occurred in the video. In order to verify the accuracy of the prediction results, it is also necessary to compare the predicted event code with the pre-defined standard event code.

[0071] Optionally, in this process, the quality of the current performance of the model can be judged by setting preset matching conditions. Since the event code identifies the event type and the time of occurrence of the event, the preset matching conditions not only need to verify the consistency between the predicted event code and the standard event code in the event type, but also need to verify whether the two are highly consistent in terms of the trigger time, duration, etc. of the event. Specifically, the preset matching conditions may require that the event types of the predicted event code and the standard event code must be consistent, and the trigger time error does not exceed 200 milliseconds. Then, by comparing the predicted event code with the standard event code, it can be determined whether the predicted event code and the standard event code meet the preset matching conditions.

[0072] Optionally, through strict preset matching conditions, the predicted event code is further compared with the standard event code of the key fragment material to determine whether the predicted event code and the standard event code meet the preset matching conditions, and the current optimization results of the event tracking model are verified by the matching results between the two. If it is found that the predicted event code and the standard event code do not meet the preset matching conditions, it means that there is a large deviation between the two. At this time, it is necessary to adjust the parameters of the initial event tracking model. The parameter adjustment calculation can be specifically realized through the similarity and gap between the predicted event code and the standard event code. The prediction ability of the model after parameter adjustment can be optimized to a certain extent, and then training and adjustment are continued based on the model after parameter adjustment. Through this cyclic process, the output accuracy of the initial event tracking model is continuously improved until the predicted event code and the standard event code output by it meet the preset matching conditions, and the model optimization is successful. Among them, the specific process of parameter adjustment is to calculate the prediction loss value according to the predicted event code and the standard event code, and then adjust the specific parameters in the initial event tracking model according to the predicted loss value.

[0073] S406: When the predicted event code output by the initial event tracking model for the original film source meets a preset matching condition with the standard event code, the image acquisition device obtains an optimized event tracking model.

[0074] Optionally, when the predicted event code and the standard event code meet the preset matching conditions, it can be considered that the gap between the predicted event code and the standard event code is already very small. At this time, the state of the optimized event tracking model obtained can already handle the problems of the original film source scene better, which means that the model has been successfully optimized this time.

[0075] In an embodiment of the present application, a method for optimizing a directed video is provided. The model is iteratively trained multiple times through a real film source with a standard label, so that the converged model can accurately identify various key events in the original film source. Before training the initial event tracking model, the key clip materials in the original film source are pre-cut out, and standard event codes are added to the selected key clip materials in a targeted manner. These key clip materials are used as samples for subsequent model training to improve the efficiency of model training. When optimizing the model, the original film source is replaced with the data stream that the simulated image acquisition device is collecting in real time, so that the image acquisition device regenerates event codes for the film source with problems. If it is found that the predicted event code and the standard event code do not meet the preset matching conditions, the parameters of the event tracking model are adjusted. Through multiple iterations of repetition, the output accuracy of the event tracking model is continuously improved until the predicted event code output by it meets the preset matching conditions with the standard event code, and an optimized event tracking model that is successfully optimized is obtained.

[0076] See also Figure 6 , Figure 6 This is a structural block diagram of a video director optimization device provided in an embodiment of the present application. Figure 6 As shown, the director video optimization device 600 is applied to the image acquisition device, and the director video optimization device 600 includes:

[0077] The data acquisition module 610 is used to determine the original film source corresponding to the current directed film.

[0078] The model output module 620 is used to input the original clip into the optimized event tracking model to obtain the optimized event code output by the optimized event tracking model; the optimized event tracking model is obtained by training the initial event tracking model according to the predicted event code and standard event code of the original film source, the standard event code is used to identify the standard key events and the triggering time of the standard key events in the original film source, and the predicted event code is the prediction result of the initial event tracking model output for the original film source.

[0079] The film repair module 630 is used to send an optimization event code to the broadcast device, and the optimization event code is used to generate an optimized broadcast film corresponding to the original film source.

[0080] Optionally, the film repair module 630 is also used to capture a close-up image of the target object in the original film source according to the optimized event code, and send the close-up image to the guide device, and the close-up image is used to generate an optimized guide film corresponding to the original film source.

[0081] Optionally, the director video optimization device 600 also includes: a model training module, which is used to obtain the standard event code of the original film source, and use the original film source as a sample in the image acquisition device simulation environment to train the initial event tracking model; when the predicted event code output by the initial event tracking model for the original film source meets the preset matching conditions with the standard event code, an optimized event tracking model is obtained.

[0082] Optionally, the model training module is also used to determine the key segment material corresponding to at least one abnormal picture in the current directed film in the original film source, and obtain the standard event code corresponding to each key segment material; use each key segment material as a sample in the image acquisition device simulation environment to train the initial event tracking model.

[0083] Optionally, the model training module is also used to store the original film source in the input data stream file of the initial event tracking model in the image acquisition device simulation environment; determine the predicted event code output by the initial event tracking model for the original film source, and judge whether the predicted event code and the standard event code meet the preset matching conditions; if not, adjust the parameters of the initial event tracking model based on the predicted event code and the standard event code.

[0084] Optionally, the director video optimization device 600 also includes: a model deployment module, used to obtain an optimized event tracking model; and replace the initial event tracking model with the optimized event tracking model.

[0085] Optionally, the data acquisition module 610 is further used to determine that the film source corresponding to the image acquisition device among all the film sources corresponding to the current directed film is the original film source.

[0086] In summary, the director video optimization device re-outputs event codes for the original film source based on the optimized model, so that the director equipment generates optimized director films according to the optimized event codes, thereby effectively repairing the current director films with poor performance and solving the problem of poor current film effects.

[0087] An embodiment of the present application further provides a computer-readable storage medium, which may store a computer program code. When the computer program code runs on an image acquisition device, the image acquisition device executes the steps of any method in the above embodiments.

[0088] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an image acquisition device provided in an embodiment of the present application. Figure 7 As shown, the image acquisition device 700 may include: at least one processor 701 , at least one network interface 704 , a user interface 703 , a memory 705 , and at least one communication bus 702 .

[0089] The communication bus 702 is used to realize the connection and communication between these components.

[0090] The user interface 703 may include a display screen (Display) and a camera (Camera), and the optional user interface 703 may also include a standard wired interface and a wireless interface.

[0091] The network interface 704 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0092] Among them, the processor 701 may include one or more processing cores. The processor 701 uses various interfaces and lines to connect various parts within the entire image acquisition device 700, and executes various functions and processes data of the image acquisition device 700 by running or executing instructions, programs, code sets or instruction sets stored in the memory 705, and calling data stored in the memory 705. Optionally, the processor 701 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 701 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 701, and can be implemented separately through a chip.

[0093] Among them, the memory 705 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 705 includes a non-transitory computer-readable storage medium. The memory 705 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 705 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 705 may also be optionally at least one storage device located away from the aforementioned processor 701. As Figure 7 As shown, the memory 705 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a director video optimization program.

[0094] exist Figure 7 In the image acquisition device 700 shown, the user interface 703 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 701 can be used to call the director video optimization program stored in the memory 705, and specifically perform the following operations:

[0095] Determine the original source corresponding to the current directed film; input the original clip into the optimized event tracking model to obtain the optimized event code output by the optimized event tracking model; the optimized event tracking model is trained by using the initial event tracking model according to the predicted event code and standard event code of the original source, the standard event code is used to identify the standard key events and the triggering time of the standard key events in the original source, and the predicted event code is the prediction result of the initial event tracking model for the original source output; send the optimized event code to the directed film device, and the optimized event code is used to generate the optimized directed film corresponding to the original source.

[0096] In some embodiments, the processor 701 also specifically performs the following steps when executing: extracting a close-up picture of the target shooting object in the original film source according to the optimization event code, and sending the close-up picture to the guidance device, and the close-up picture is used to generate an optimized guidance film corresponding to the original film source.

[0097] In some embodiments, the processor 701 also specifically performs the following steps: obtaining the standard event code of the original film source; using the original film source as a sample in the image acquisition device simulation environment to train the initial event tracking model; when the predicted event code output by the initial event tracking model for the original film source meets the preset matching condition with the standard event code, an optimized event tracking model is obtained.

[0098] In some embodiments, when the processor 701 executes the acquisition of the standard event code of the original film source, it specifically performs the following steps: determining the key segment material corresponding to at least one abnormal scene in the current directed film in the original film source, and obtaining the standard event code corresponding to each key segment material; when the image acquisition device processor 701 executes the training of the initial event tracking model using the original film source as a sample in the image acquisition device simulation environment, it specifically performs the following steps: using each key segment material as a sample in the image acquisition device simulation environment to train the initial event tracking model.

[0099] In some embodiments, when the processor 701 uses the original film source as a sample to train the initial event tracking model in the image acquisition device simulation environment, it specifically performs the following steps: in the image acquisition device simulation environment, the original film source is stored in the input data stream file of the initial event tracking model; the predicted event code output by the initial event tracking model for the original film source is determined, and it is determined whether the predicted event code and the standard event code meet the preset matching conditions; if not, the parameters of the initial event tracking model are adjusted based on the predicted event code and the standard event code.

[0100] In some embodiments, after obtaining the optimized event tracking model, the processor 701 further specifically performs the following steps: obtaining the optimized event tracking model; and replacing the initial event tracking model with the optimized event tracking model.

[0101] In some embodiments, when the processor 701 determines the original film source corresponding to the current directed film, it specifically performs the following steps: determining that the film source corresponding to the image acquisition device 700 among all the film sources corresponding to the current directed film is the original film source.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0103] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, the above process or function according to the embodiment of this specification is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions can be stored in a computer-readable storage medium or transmitted by the above computer-readable storage medium. The above computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The above computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The above-mentioned available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, digital versatile discs (DVD)), or semiconductor media (for example, solid state drives (SSD)), etc.

[0105] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0106] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0107] The above is a description of a director video optimization method, device, storage medium and image acquisition device provided by the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for optimizing a broadcast video, characterized in that: Applied to an image acquisition device, the method comprises: Determine the original film source corresponding to the current directed film; The original clip is input into an optimized event tracking model to obtain an optimized event code output by the optimized event tracking model; the optimized event tracking model is obtained by training the initial event tracking model according to the predicted event code and the standard event code of the original film source, the standard event code is used to identify the standard key event in the original film source and the triggering time of the standard key event, and the predicted event code is the prediction result output by the initial event tracking model for the original film source; The optimization event code is sent to the broadcast guidance device, and the optimization event code is used to generate an optimized broadcast guidance film corresponding to the original film source.

2. The method according to claim 1, characterized in that The method further comprises: Extracting a close-up image of the target object in the original film source according to the optimized event code; The close-up picture is sent to the director device, and the close-up picture is used to generate an optimized director film corresponding to the original film source.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtaining a standard event code of the original film source; Using the original film source as a sample in an image acquisition device simulation environment to train the initial event tracking model; When the predicted event code and the standard event code meet a preset matching condition, the optimized event tracking model is obtained.

4. The method according to claim 3, characterized in that: The step of obtaining the standard event code of the original film source includes: Determine the key segment material corresponding to at least one abnormal scene in the current directed film in the original film source, and obtain the standard event code corresponding to each key segment material; The method of using the original film source as a sample in the image acquisition device simulation environment to train the initial event tracking model includes: The key fragment materials are used as samples in an image acquisition device simulation environment to train the initial event tracking model.

5. The method according to claim 3, characterized in that: The method of using the original film source as a sample in the image acquisition device simulation environment to train the initial event tracking model includes: In an image acquisition device simulation environment, storing the original film source into an input data stream file of the initial event tracking model; Determine the predicted event code output by the initial event tracking model for the original film source, and judge whether the predicted event code satisfies a preset matching condition with the standard event code; If not, the parameters of the initial event tracking model are adjusted based on the predicted event code and the standard event code.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtaining the optimized event tracking model; The initial event tracking model is replaced with the optimized event tracking model.

7. The method according to claim 1 or 2, characterized in that: The determining of the original film source corresponding to the current directed film includes: Determine that the film source corresponding to the image acquisition device among all the film sources corresponding to the currently directed film is the original film source.

8. A video director optimization device, characterized in that: Applied to an image acquisition device, the device comprises: A data acquisition module is used to determine the original film source corresponding to the current directed film; A model output module is used to input the original clip into an optimized event tracking model to obtain an optimized event code output by the optimized event tracking model; the optimized event tracking model is obtained by training the initial event tracking model according to the predicted event code and the standard event code of the original film source, the standard event code is used to identify the standard key event in the original film source and the triggering time of the standard key event, and the predicted event code is the prediction result output by the initial event tracking model for the original film source; The film repair module is used to send the optimization event code to the guide device, and the optimization event code is used to generate an optimized guide film corresponding to the original film source.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program codes, and when the computer program codes are executed on an image acquisition device, the image acquisition device is enabled to execute the steps of the method according to any one of claims 1 to 7.

10. An image acquisition device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.

Images