Methods for invoking algorithms, electronic devices, and readable storage media

Abstract

This application discloses a method for invoking an algorithm, an electronic device, and a readable storage medium, belonging to the field of terminal technology. It includes: a media platform framework layer receiving a first instruction sent by a first application, the first instruction including address indication information of the material to be analyzed. In response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed based on the address indication information, and sends a second instruction to an algorithm service framework layer, the second instruction including the material data and target identification information. In response to the second instruction, the algorithm service framework layer, based on the target identification information, performs highlight segment analysis of the material to be analyzed based on the material data through a first service call. This application provides an interface for algorithm invocation to the outside world through the media platform framework layer, and provides services for interfacing with different algorithms within the algorithm service framework layer, enabling any application to call the underlying algorithm for material processing.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a method for invoking an algorithm, an electronic device, and a readable storage medium. Background Technology

[0002] With the rapid development of terminal technology, various applications are provided in electronic devices to meet different user business needs. Some applications may need to use underlying chip capabilities during business implementation, including chip-level algorithms and hardware acceleration functions. For example, in the one-click video creation service, it is usually necessary to call the underlying highlight fragment algorithm to analyze the highlights of the material to be analyzed, that is, to identify the highlights in the material to be analyzed. This allows the application to filter, crop, and merge the material to be analyzed based on the highlights fragment analysis results, in order to synthesize a wonderful video clip.

[0003] However, in the current system architecture, there is a problem that third-party applications cannot access the underlying capabilities of the chip. Summary of the Invention

[0004] This application provides a method for invoking an algorithm, an electronic device, and a readable storage medium, which can solve the problem in related technologies where third-party applications cannot access the underlying capabilities of the chip. The technical solution is as follows:

[0005] Firstly, a method for invoking an algorithm is provided, applied to an electronic device. The electronic device includes a first application, a media middleware framework layer, an algorithm service framework layer, and an algorithm implementation layer. The algorithm service framework layer provides multiple services, each of which interfaces with a chip-level capability in the algorithm implementation layer. The method includes:

[0006] The media platform framework layer receives a first instruction sent by the first application, the first instruction including address indication information of the material to be analyzed;

[0007] In response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed based on the address indication information;

[0008] The media middleware framework layer sends a second instruction to the algorithm service framework layer. The second instruction includes the material data and target identification information. The target identification information is used to identify the instruction related to the current business request of the first application.

[0009] In response to the second instruction, the algorithm service framework layer invokes the highlight fragment algorithm through the first service based on the target identification information, so as to perform highlight fragment analysis on the material to be analyzed based on the material data using the highlight fragment algorithm.

[0010] The material to be analyzed is the target material selected by the user. In one example, the material to be analyzed may include multiple target video materials, or multiple target image materials, or multiple target video materials and multiple target image materials.

[0011] For example, the first instruction is the material analysis instruction r7.

[0012] For example, the address indication information is a file descriptor. In one possibility, the media data is the frame data address of the target media in the shared buffer; in another possibility, the media data could also be the decoded data of the target media.

[0013] For example, the target identification information is identifier a in the specific embodiments below. Different identification information can be set for different services.

[0014] In practice, the second instruction varies depending on the materials included in the material to be analyzed. As an example of this application, when the material to be analyzed includes multiple target image materials, the second instruction includes an image analysis instruction. As another example of this application, when the material to be analyzed includes multiple target video materials, the second instruction includes a video analysis instruction.

[0015] As an example of this application, when the material to be analyzed includes multiple target video materials, the first application can determine the total target analysis duration of the multiple target video materials from the media platform framework layer before sending the first instruction to the media platform framework layer. In this case, the first instruction can also carry the total target analysis duration so that the media platform framework layer can select the corresponding analysis strategy for each target video material according to the total target analysis duration.

[0016] Optionally, the first application may not determine the total target analysis duration of multiple target video clips. In this case, in subsequent processes, the media platform framework layer can assign corresponding analysis strategies to each target video clip according to preset strategies, such as random allocation or allocation based on the video duration of each target video clip.

[0017] As an example of this application, the target identification information is bound to the first service. Therefore, the corresponding first service can be determined based on the target identification information, and the first service can be called, thereby calling the highlight fragment algorithm through the first service.

[0018] In this way, by providing interfaces for algorithm calls to the outside world through the media middle platform framework layer, and providing services for interfacing with different algorithms in the algorithm service framework layer, any application can call the underlying algorithms to process materials.

[0019] As an example of this application, the algorithm service framework layer includes a service interface layer, a service transport layer, a service management layer, and a service function layer. The service interface layer includes at least one service interface module, and the service function layer includes at least one service function module. Each service interface module corresponds to a chip underlying capability, and each service function module interfaces with a chip underlying capability.

[0020] The media middleware framework layer sends a second instruction to the algorithm service framework layer, including:

[0021] The media platform framework layer calls the first service interface module in the service interface layer that corresponds to the highlight fragment algorithm, and sends the second instruction to the material analysis interface of the first service interface module;

[0022] In response to the second instruction, the algorithm service framework layer, based on the target identification information, invokes the highlight fragment algorithm through the first service to perform highlight fragment analysis on the material to be analyzed based on the material data, including:

[0023] In response to the second instruction, the first service interface module sends a third instruction to the service transport layer through the material analysis interface. The third instruction carries the material data, the target identification information, the first module identifier of the first service interface module, and the interface indication information of the material analysis interface.

[0024] The service transport layer sends the third instruction to the service management layer;

[0025] In response to the third instruction, the service management layer, based on the target identification information, the first module identification, and the interface instruction information of the material analysis interface, calls the highlight fragment algorithm through the first service function module in the service function layer that interfaces with the highlight fragment algorithm, so as to perform highlight fragment analysis on the material to be analyzed based on the material data.

[0026] For example, the first service interface module is service interface module 2 in the embodiments below. The first service function module is service function module 2 in the embodiments below. Both the first service interface module and the first service function module are modules corresponding to the specular fragment algorithm.

[0027] Optionally, the media analysis interface may include an image analysis interface or a video analysis interface. For example, the media analysis interface called when a request is made to analyze target image media is an image analysis interface; similarly, the media analysis interface called when a request is made to analyze target video media is a video analysis interface.

[0028] Interface indication information is used to indicate the interface type of the interface being called.

[0029] In one example, the third instruction is a video frame analysis instruction. That is, when the media middle platform framework layer requests frame-by-frame analysis of the material to be analyzed, the third instruction can be sent to the first service interface module. The third instruction carries the material data of the current video frame to be analyzed, the target identification information, the first module identification of the first service interface module, and the interface indication information of the material analysis interface (such as video analysis interface or image analysis interface).

[0030] In this way, by providing service interface modules corresponding to different algorithms that interact with the media platform framework layer in the algorithm service framework layer, and by providing service function modules corresponding to different algorithms that interact with the algorithm implementation layer in the algorithm service framework layer, the algorithm service framework layer can provide services corresponding to different algorithms. This allows applications to use the underlying algorithms by calling different services of the algorithm service framework layer through the media platform framework layer, thereby improving the openness of the algorithm to the outside world.

[0031] As an example of this application, in response to the third instruction, the service management layer, based on the target identification information, the first module identifier, and the interface indication information of the material analysis interface, calls the highlight fragment algorithm through the first service function module in the service function layer that interfaces with the highlight fragment algorithm, so as to perform highlight fragment analysis on the material to be analyzed based on the material data by the highlight fragment algorithm, including:

[0032] In response to the third instruction, the service management layer determines the first service execution instance corresponding to the first service function module from the service function layer based on the target identification information, the first module identification, and the target binding relationship. The first service execution instance is a service execution instance that supports the first application in calling the highlight fragment algorithm. The target binding relationship includes identification information, module identification, and the binding relationship between service execution instances.

[0033] The service management layer invokes the first service execution instance of the service function layer to send the interface instruction information of the material analysis interface and the material data to the first service execution instance;

[0034] The service function layer transmits the material data to the highlight fragment algorithm through the first service execution instance according to the interface instruction information, so that the highlight fragment algorithm performs highlight fragment analysis on the material to be analyzed based on the material data.

[0035] In one example, the first service execution instance is service execution instance T2 in the following embodiment.

[0036] Thus, within the algorithm service framework layer, different business operations can interact with the algorithm implementation layer through different service execution instances corresponding to the service. This allows multiple business operations to use the same service in the algorithm service framework layer in parallel, enabling multiple business operations to simultaneously call a specific algorithm in the underlying algorithm implementation layer.

[0037] As an example of this application, before the media middleware framework layer receives the first instruction sent by the first application, it further includes:

[0038] The first application receives a user's selection of the material to be analyzed and confirmation to execute the one-click video creation function, which is used to generate a video clip including highlight segments based on the material to be analyzed.

[0039] In response to the operation, the first application calls the initialization interface related to the one-click video creation function in the media middleware framework layer;

[0040] The media middleware framework layer generates the target identification information and sends an algorithm initialization instruction to the first service in the algorithm service framework layer. The algorithm initialization instruction carries the target identification information.

[0041] The algorithm service framework layer initializes the highlight fragment algorithm through the first service;

[0042] The algorithm service framework layer binds the target identification information to the first service.

[0043] Thus, during the algorithm initialization phase, by binding the target identification information with the first service, since the target identification information is used to represent the request related to the current business of the first application, that is, it is the unique identification information assigned to the current business of the first application, the algorithm can then interact with the highlight fragment algorithm through the dedicated first service, thereby improving the independence of algorithm calls.

[0044] As an example of this application, the media middleware framework layer includes a design interface layer and a channel interface layer, wherein the design interface layer includes the initialization interface;

[0045] The media middleware framework layer sends an algorithm initialization instruction to the first service in the algorithm service framework layer, including:

[0046] The design interface layer sends a fourth instruction to the channel interface layer, the fourth instruction including the algorithm name of the highlight fragment algorithm and the interface indication information of the initialization interface of the first service interface module;

[0047] In response to the fourth instruction, the channel interface layer calls the initialization interface in the first service interface module and sends the algorithm initialization instruction to the initialization interface of the first service interface module according to the algorithm name and the interface indication information of the initialization interface of the first service interface module.

[0048] In one example, the fourth instruction carries initialization parameters sent down from the design interface layer. For example, the initialization parameters include the algorithm name of the highlight fragment algorithm and the interface indication information of the initialization interface of the first service interface module.

[0049] In one example, the algorithm initialization instruction is the initialization instruction r3, which is used to request the initialization of the specular fragment algorithm.

[0050] Thus, the channel interface layer determines that the first service interface module corresponding to the highlight fragment algorithm needs to be called based on the algorithm name, and that the highlight fragment algorithm needs to be initialized based on the interface indication information of the initialization interface of the first service interface module. Therefore, the initialization interface of the first service interface module is called, and the algorithm initialization instruction is sent to the initialization interface of the first service interface module to request the initialization of the highlight fragment algorithm.

[0051] As an example of this application, the algorithm service framework layer initializes the highlight fragment algorithm through the first service, including:

[0052] The first service interface module sends a fifth instruction to the service transport layer. The fifth instruction includes the target identification information, the first module identifier, and the interface indication information of the initialization interface of the first service interface module.

[0053] The service transport layer sends the fifth instruction to the service management layer;

[0054] The service management layer creates a first service execution instance corresponding to the first service function module based on the first module identifier;

[0055] The service management layer invokes the first service execution instance and sends the interface indication information of the initialization interface of the first service interface module to the first service execution instance;

[0056] The service function layer initializes the first service execution instance, and the first service execution instance initializes the highlight fragment algorithm according to the interface instruction information of the initialization interface of the first service interface module.

[0057] In one example, the fifth instruction is the initialization instruction r3 after the first module identifier has been added.

[0058] To facilitate lower-level modules in identifying which service interface module and which interface of the service interface layer has been called, the first service interface module sends a fifth instruction to the service management layer through the service transport layer. This fifth instruction carries target identification information, the first module identifier, and the interface indication information of the first service interface module's initialization interface. In this way, the service management layer can create a first service execution instance corresponding to the target identifier information based on the first module identifier. This facilitates subsequent interactions between the service function layer and the highlight fragment algorithm through the first service execution instance for other requests from the first application's current business logic. Consequently, the call to the highlight fragment algorithm by the first application's current business logic can be independent of other business logic.

[0059] As an example of this application, the algorithm service framework layer binds the target identification information to the first service, including:

[0060] The service management layer binds the target identification information, the first module identification, and the first service execution instance.

[0061] In this way, the target identification information, the first module identification, and the first service execution instance are bound together so that subsequent calls to the highlight fragment algorithm for this business of the first application can determine the first service execution instance based on the target identification information and the first module identification, and thus realize the call through the first service execution instance.

[0062] As an example of this application, in response to the second instruction, the algorithm service framework layer, based on the target identification information, calls the highlight fragment algorithm through the first service to perform highlight fragment analysis on the material to be analyzed based on the material data using the highlight fragment algorithm, and then further includes:

[0063] The algorithm service framework layer obtains the highlight fragment analysis results through the first service;

[0064] The algorithm service framework layer sends the highlight fragment analysis results to the media middleware framework layer;

[0065] The media middleware framework layer sends the highlight clip analysis results to the first application.

[0066] The first application edits the material to be analyzed based on the highlight fragment analysis results to generate the video fragment.

[0067] In one example, when the material to be analyzed includes multiple target video clips, the highlight fragment analysis result includes the highlight fragment location information of each target video clip. When the material to be analyzed also includes multiple target image clips, the highlight fragment analysis result also includes the score value of each target image clip.

[0068] As an example of this application, the material to be analyzed includes target video material; the algorithm service framework layer obtains the highlight segment analysis results through the first service, including:

[0069] The service function layer receives the highlight fragment analysis results through the first service execution instance;

[0070] The algorithm service framework layer sends the highlight segment analysis results to the media middleware framework layer, including:

[0071] The service function layer sends the highlight fragment analysis results to the service management layer;

[0072] The service management layer calls back to the first service interface module through the transport layer to send the highlight fragment analysis results to the first service interface module;

[0073] The first service interface module sends the highlight fragment analysis results to the media platform framework layer.

[0074] In this way, the highlight fragment analysis results are received through the first service execution instance, and the highlight fragment analysis results are fed back to the first service interface module through a callback. This enables the first service interface module to send the highlight fragment analysis results to the media platform framework layer, thereby realizing the analysis of the material to be analyzed through the highlight fragment algorithm.

[0075] As an example of this application, in response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed according to the address indication information, including:

[0076] In response to the first instruction, the media platform framework layer retrieves the material to be analyzed from the corresponding storage address according to the address indication information;

[0077] The media platform framework layer stores the material to be analyzed in a shared cache, which is a cache shared by the media platform framework layer and the algorithm implementation layer.

[0078] The media platform framework layer determines the frame data address of the material to be analyzed in the shared cache, and the material data is the frame data address of the material to be analyzed in the shared cache.

[0079] In this way, by storing the material to be analyzed in a shared cache and determining the frame data address in the shared cache as the material data, the frame data address can be transmitted during subsequent module interactions instead of directly transmitting the material to be analyzed, thereby reducing the amount of data transmitted between modules.

[0080] As an example of this application, in response to the first instruction, after the media platform framework layer obtains the material to be analyzed from the corresponding storage address according to the address indication information, it further includes:

[0081] The media platform framework layer performs resolution reduction processing on the material to be analyzed to obtain low-resolution material;

[0082] The media middleware framework layer stores the media to be analyzed in a shared cache, including:

[0083] The media middleware framework layer stores the low-resolution materials in the shared cache;

[0084] The media platform framework layer determines the frame data address of the material to be analyzed in the shared cache, wherein the material data is the frame data address of the material to be analyzed in the shared cache, including:

[0085] The media platform framework layer determines the frame data address of the low-resolution material in the shared cache, and the material data is the frame data address of the low-resolution material in the shared cache.

[0086] Therefore, by reducing the resolution of the material to be analyzed before requesting the specular fragment algorithm to be used for analysis, the analysis efficiency of subsequent specular fragment analysis can be improved.

[0087] As an example of this application, in response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed according to the address indication information, including:

[0088] In response to the first instruction, the media platform framework layer retrieves the material to be analyzed from the corresponding storage address according to the address indication information;

[0089] The media platform framework layer stores the material to be analyzed in a shared cache, which is a cache shared by the media platform framework layer and the algorithm implementation layer.

[0090] The media middleware framework layer sends a material compression instruction to the algorithm service framework layer. The material compression instruction carries the frame data address of the material to be analyzed in the shared cache and the target identification information.

[0091] In response to the material compression instruction, the algorithm service framework layer, based on the target identification information, calls the resolution reduction algorithm of the algorithm implementation layer through the second service, so as to perform resolution reduction processing on the material to be analyzed based on the frame data address of the material to be analyzed in the shared cache.

[0092] The algorithm service framework layer obtains the frame data address of the low-resolution material in the shared cache through the second service. The down-resolution material is the data after the down-resolution processing of the material to be analyzed is performed by the down-resolution algorithm.

[0093] The algorithm service framework layer sends the frame data address of the low-resolution material in the shared cache to the media platform framework layer;

[0094] The media middleware framework layer determines the frame data address of the low-resolution material in the shared cache as the material data.

[0095] The media compression command requests that the media to be analyzed be down-resolution processed. Thus, by calling the down-resolution algorithm implemented in the algorithm layer, the media to be analyzed is down-resolution processed. Afterwards, the resulting low-resolution media is stored in a shared buffer, and the frame data address of the low-resolution media in the shared buffer is passed to the highlight fragment algorithm. This allows the highlight fragment algorithm to perform highlight fragment analysis on the low-resolution media, improving the analysis efficiency of the highlight fragment algorithm.

[0096] As an example of this application, the media middleware framework layer sends a material compression instruction to the algorithm service framework layer, including:

[0097] The media platform framework layer calls the second service interface module in the service interface layer that corresponds to the resolution reduction algorithm, and sends the material compression command to the resolution reduction interface of the second service interface module;

[0098] In response to the material compression command, the algorithm service framework layer, based on the target identification information, calls the resolution reduction algorithm of the algorithm implementation layer through the second service to perform resolution reduction processing on the material to be analyzed based on the frame data address of the material to be analyzed in the shared cache, including:

[0099] In response to the material compression command, the second service interface module sends a sixth command to the service transport layer through the resolution reduction interface. The sixth command carries the frame data address of the material to be analyzed in the shared cache, the target identification information, the second module identifier of the second service interface module, and the interface indication information of the resolution reduction interface.

[0100] The service transport layer sends the sixth instruction to the service management layer;

[0101] In response to the sixth instruction, the service management layer, based on the target identification information, the second module identification, and the interface indication information of the resolution reduction interface, calls the resolution reduction algorithm through the second service function module in the service function layer that interfaces with the resolution reduction algorithm, so as to perform resolution reduction processing on the material to be analyzed based on the frame data address of the material to be analyzed in the shared cache.

[0102] In one example, the second service interface module is service interface module 1 in the following embodiment, and the second service function module is service function module 1 in the following embodiment.

[0103] In one example, the sixth instruction is a video frame compression instruction or an image compression instruction in the embodiments below.

[0104] In this way, the second service is implemented through the second service interface module and the second service function module, and the interaction with the resolution reduction algorithm is achieved through the second service, which can isolate the resolution reduction algorithm from other business calls.

[0105] As an example of this application, before the media middleware framework layer sends the material compression instruction to the algorithm service framework layer, it further includes:

[0106] The media middleware framework layer sends an algorithm initialization instruction to the second service in the algorithm service framework layer, and the algorithm initialization instruction carries the target identification information.

[0107] The algorithm service framework layer initializes the resolution reduction algorithm through the second service;

[0108] The algorithm service framework layer binds the target identification information to the second service.

[0109] As an example of this application, the media platform framework layer includes a highlight segment analysis interface and a policy monitoring interface; the media platform framework layer receives a first instruction sent by the first application, including:

[0110] The first application sends the first instruction to the highlight fragment analysis interface;

[0111] The highlight fragment analysis interface sends the first instruction to the strategy monitoring interface;

[0112] In response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed based on the address indication information, including:

[0113] In response to the first instruction, the strategy monitoring interface obtains the material data of the material to be analyzed based on the address indication information.

[0114] Thus, the media platform framework layer provides a highlight segment analysis interface, allowing the first application to request highlight segment analysis from the algorithm implementation layer by calling this interface. In other words, the media platform framework layer provides an open interface, enabling applications to invoke the underlying algorithms through these calls.

[0115] Secondly, an apparatus for invoking an algorithm is provided, the apparatus having the functionality to implement the method behavior of invoking the algorithm as described in the first aspect. The apparatus for invoking the algorithm includes at least one module, the at least one module being used to implement the method of invoking the algorithm provided in the first aspect.

[0116] Thirdly, an electronic device is provided, comprising a processor and a memory. The memory stores a program that supports the electronic device in executing the calling algorithm provided in the first aspect, and stores data related to the method for implementing the calling algorithm described in the first aspect. The processor is configured to execute the program stored in the memory. The electronic device may further include a communication bus for establishing a connection between the processor and the memory.

[0117] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the method of the calling algorithm described in the first aspect.

[0118] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the calling algorithm described in the first aspect above.

[0119] The technical effects achieved by the second, third, fourth, and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0120] Figure 1 This is a schematic diagram illustrating an application scenario according to an exemplary embodiment;

[0121] Figure 2 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0122] Figure 3This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0123] Figure 4 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0124] Figure 5 This is a schematic diagram of the framework of a software system for an electronic device according to an exemplary embodiment;

[0125] Figure 6 This is a flowchart illustrating a method for invoking an algorithm according to an exemplary embodiment;

[0126] Figure 7 This is a schematic diagram of the framework of a software system for an electronic device according to another exemplary embodiment;

[0127] Figure 8 This is a flowchart illustrating a method for invoking an algorithm according to another exemplary embodiment;

[0128] Figure 9 This is a flowchart illustrating a method for invoking an algorithm according to another exemplary embodiment;

[0129] Figure 10 This is a flowchart illustrating a method for invoking an algorithm according to another exemplary embodiment;

[0130] Figure 11 This is a flowchart illustrating a method for invoking an algorithm according to another exemplary embodiment;

[0131] Figure 12 This is a flowchart illustrating a method for invoking an algorithm according to another exemplary embodiment;

[0132] Figure 13 This is a schematic diagram of the frame of an electronic device according to an exemplary embodiment. Detailed Implementation

[0133] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0134] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0135] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0136] In an exemplary application scenario, a user pre-captures multiple video clips using their mobile phone, and the phone stores these video clips in its gallery. For example... Figure 1 As shown in Figure (a), a gallery icon is displayed on the phone's home screen. When a user wants to create an edited video clip based on multiple video clips, they can click the gallery icon on the home screen. In response to the user's click on the icon, the phone displays the gallery interface A1, as shown... Figure 1 As shown in Figure (b), the gallery interface A1 provides a "One-Click Blockbuster" control, such as... Figure 1 As shown in Figure (c), the user can click the "One-Click Blockbuster" control. In response to the user's click on the "One-Click Blockbuster" control, the phone displays the gallery interface A2, as shown... Figure 1 As shown in Figure (d), the gallery interface A2 can include multiple recently captured video clips, such as those captured on November 10, 2022. Users can click on any one of the multiple video clips, such as... Figure 1As shown in Figure (e), when a user clicks on the first video clip among multiple video clips, the phone displays the gallery interface A3 in response to the user's click on any video clip. Gallery interface A3 includes all the clips in the gallery. Thus, the user can select the video clips needed to generate the video segment within this gallery interface A3, for example... Figure 1 As shown in Figure (f), suppose the user selects 5 video clips. The gallery interface A3 also includes video generation options, such as... Figure 1 As shown in Figure (f), the video generation option is a checkmark control, also known as a "√" control. Please refer to [the relevant documentation]. Figure 2 In Figure (a), after selecting video footage, the user can click on the video generation option, such as clicking the checkmark. Figure 2 As shown in Figure (b), in response to the user's click on the checkmark control, the phone begins analyzing the five video clips selected by the user in the gallery interface A3. It selects highlight clips (i.e., exciting clips, such as clips of the user smiling or winning moments) from each video clip and generates video segments based on these selected highlights. During this process, the phone can display the clip analysis progress in the gallery interface A3, allowing the user to visually monitor the analysis progress. In one example, such as... Figure 3 As shown in Figure (a), after the video clip is generated, the phone displays the gallery interface A4, which includes the generated video clip. The phone can automatically play the video clip. Furthermore, the gallery interface A4 provides video export options, such as... Figure 3 As shown in Figure (b), the user can click the video export option. In response to the user's click on the video export option, the phone exports the video clip, as shown in Figure (b). Figure 3 As shown in Figure (c), the video clip is stored in the gallery, so that users can view the video clip from the gallery.

[0137] In another exemplary application scenario, a user pre-captures multiple video clips and multiple image clips using their mobile phone, which then stores these clips in its gallery. The user can then select video clips and image clips from the gallery, allowing the phone to generate video clips based on these clips. In one example, such as... Figure 4 As shown in Figure (a), after entering the gallery interface A2, gallery interface A2 includes multiple recently captured video clips and multiple image clips. The user can click on any of the video clips and image clips, for example, clicking on the first video clip. In response to the user's click on any clip, the phone displays gallery interface A3, which includes all the clips in the gallery. Thus, the user can select the clips needed to generate a video clip from gallery interface A3, for example... Figure 4As shown in Figure (b), suppose the user selected 4 video clips and 2 image clips. The gallery interface A3 also includes video generation options, such as... Figure 4 As shown in Figure (b), the video generation option is checked. Please refer to... Figure 4 In Figure (c), after selecting materials, the user can click on the video generation option, such as clicking the checkmark. Figure 4 As shown in Figure (d), in response to the user's click on the checkmark control, the phone begins analyzing the four video clips and two image clips selected by the user in the gallery interface A3. It selects highlight segments from each video clip and highlight images (i.e., best images) from the two image clips. During this process, the phone can display the analysis progress in the gallery interface A3, allowing the user to visually monitor the progress. In one example, such as... Figure 3 As shown in Figure (a), after the video clip is generated, the phone displays the gallery interface A4, which includes the generated video clip. The phone can automatically play the video clip. Furthermore, the gallery interface A4 provides video export options, such as... Figure 3 As shown in Figure (b), the user can click the video export option. In response to the user's click on the video export option, the phone exports the video clip, as shown in Figure (b). Figure 3 As shown in Figure (c), the video clip is stored in the gallery, so that users can view the video clip from the gallery.

[0138] In one example, if the user selects a small number of materials, the phone can provide prompts during the selection process to help the user determine the appropriate number of materials to choose. For example... Figure 1 As shown in Figure (f), the gallery interface A3 displays the message "6 or more clips will produce better results," which helps users know at least how many clips they need to select to generate a better video clip.

[0139] In one example, if the user selects a large number of materials, the phone can provide prompts during the selection process so that the user knows the maximum number of materials they can select. For example... Figure 4 As shown in Figure (b), the gallery interface A3 displays the message "A maximum of 30 materials can be selected", which makes it easy for users to know or how many materials they can select.

[0140] In one example, after generating the video clip, other functional options can be provided in the application interface A4 displaying the video clip, allowing users to edit, add effects, analyze, and perform other operations on the generated video clip based on these options. Figure 3As shown in Figure (a), these other feature options may include, but are not limited to, templates, music, clips, sharing, and other options.

[0141] It should be noted that the above application scenario only illustrates the example where the phone displays the gallery interface A2 after receiving a click on the "One-Click Blockbuster" control in gallery interface A1. In another example, after receiving a click on the "One-Click Blockbuster" control in gallery interface A1, the phone can also directly display gallery interface A3, as shown below. Figure 1 As shown in Figure (f) in the example, this allows users to select materials based on the gallery interface A3, but this embodiment of the application does not limit this.

[0142] It should also be noted that the above application scenario is only illustrated by providing a one-click photo creation function in a photo library. In another example, the mobile phone can also provide a one-click photo creation function in applications such as video editing apps, camera apps, or other third-party applications, and the methods provided in the embodiments of this application can be used to analyze, synthesize, and process the materials.

[0143] The software system architecture of the electronic device involved in the embodiments of this application will be described next. Figure 5 This is a schematic diagram of the software structure of the electronic device provided in an embodiment of this application. For example... Figure 5 As shown, electronic devices can adopt a layered architecture, dividing the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom as follows: application (APP) layer, media platform framework layer, application framework (FWK) layer, and hardware abstraction layer (HAL).

[0144] The application layer, or application layer for short, can include a series of application packages, such as camera, gallery, calendar, map, and navigation. When these application packages are run, they can access the various modules provided by the media platform framework layer and the application framework layer through application programming interfaces (APIs) and execute corresponding intelligent business functions.

[0145] In some embodiments, the camera is used to capture images, videos, slow-motion images, and panoramic images in response to a user's shooting action. After these images are captured by the camera, or after the user triggers a screenshot, or after the user triggers screen recording, or after the electronic device downloads images from other devices, the electronic device can save these images to a gallery, allowing the user to perform video editing operations on the images in the gallery, such as one-click video editing.

[0146] In this embodiment, the image library is divided into three layers from top to bottom: video editing business layer, application function layer, and basic function layer.

[0147] The video editing service layer offers various services such as automatic multi-camera recording, AI-powered music videos, one-click video creation, and highlight reels. These services are presented as controls in the gallery interface. Users can trigger corresponding video processing actions by manipulating a control. For example... Figure 1 As shown in the interactive process, after the user clicks the "One-Click Blockbuster" control provided in the gallery interface A1, they can select video materials from the gallery interface A3, which includes multiple video materials. After clicking the checkmark control in the gallery interface A3, a video clip synthesized based on the selected video material is generated in the gallery.

[0148] The application functionality layer includes an automatic editing framework. Each business function in the business layer can invoke this framework to provide automatic editing services for image and video footage. For example, the automatic editing framework may include modules such as clip selection, storyline organization, layout splicing, and special effects enhancement. Clip selection is used to obtain highlight clips from the footage by calling underlying interfaces. Storyline organization is used to sequentially splice multiple clips in a storyline format based on the content of the footage. Layout splicing is used to adjust the interface layout of the footage. Special effects enhancement is used to adjust the video's enhancement effects, such as adjusting screen brightness and beautifying facial features.

[0149] The basic functionality layer is used to perform basic processing on the edited video clips after multiple clips have been edited by the automatic editing framework. For example, the basic functionality layer may include basic functional modules such as video splicing, compositing and saving, video effects rendering, and audio effects processing. Specifically, video splicing is used to splice multiple extracted highlight clips. Compositing and saving is used to store the spliced ​​video clips. Video effects rendering is used to add video effects to the spliced ​​video clips, such as adding style filters and themes. Audio effects processing is used to add sound effects to the spliced ​​video clips, such as adding background music.

[0150] The media middle platform framework layer is a software layer set up between the application layer and the application framework. As an example of this application, the media middle platform framework layer mainly includes a solution interface and a pipeline interface. The solution interface is used to interact with the application layer. For example, a library can request the underlying layer to process materials by calling the solution interface. The pipeline interface is used to interact with the FWK layer. That is, after the solution interface receives a call from the application layer, it sends relevant parameters to the pipeline interface. Then, the pipeline interface interacts with the FWK layer based on these parameters.

[0151] In one example, the `solution` interface includes, but is not limited to, an initialization interface, an analysis performance query interface, a highlight segment analysis interface, a strategy monitoring interface, and a topic summary interface. The initialization interface is used to initialize the algorithm. The analysis performance query interface is used to query the video analysis capabilities of the chip's underlying algorithm (such as the highlight segment algorithm). The highlight segment analysis interface is used to request analysis of highlight segments in the footage using the underlying highlight segment algorithm. The strategy monitoring interface is used to configure the available analysis duration for the video footage and, based on parameters such as the available analysis duration and the duration of each video clip, to set corresponding dynamically configured analysis strategies for the video footage. The topic summary interface is used to retrieve the topic corresponding to the content of the highlight segments.

[0152] The application framework layer, or simply the framework layer, supports the operation of various modules within the media middleware framework layer. As an example in this application, the framework layer includes an algorithm service framework layer, which provides various services, each interfacing with an underlying algorithm or capability. For example... Figure 5 As shown, the algorithm service framework layer includes a service interface layer, a service transport layer, a service management layer, and a service function layer. The service interface layer may include at least one service interface module, each corresponding to a specific algorithm; the service transport layer includes a service transport module. The service management layer includes a service management module, and the service function layer includes at least one service function module, each used to interface with a specific algorithm.

[0153] The service interface module is used to provide the media middleware framework layer with service interfaces for calling the underlying algorithms or capabilities of the chip. As an example and not a limitation, the service interface module includes an initialization interface, a parameter management interface, a resolution reduction interface, a video analysis interface, an image analysis interface, a performance analysis interface, a theme analysis interface, and a resource release interface. The interface includes the following components: Initialization Interface (for initializing the chip's underlying algorithms); Parameter Management Interface (for setting the necessary parameters for the chip's underlying algorithms, such as frame width and frame height); Downscaling Interface (for calling the underlying downscaling algorithm to reduce the resolution of the source material); Video Analysis Interface (for passing video source material data to the underlying layer for processing by the chip's underlying algorithms, such as analyzing the video source material using the underlying highlight fragment algorithm to determine highlight fragments); Image Analysis Interface (for passing image source material data to the underlying layer for processing by the chip's underlying algorithms, such as analyzing the image source material using the underlying highlight fragment algorithm to determine if the image source material belongs to a highlight fragment); Performance Analysis Interface (for querying the chip's processing capabilities, such as querying the chip's analysis speed for the highlight fragment algorithm); Theme Analysis Interface (for querying the theme of the highlight fragment); and Resource Release Interface (for releasing resources when the service terminates).

[0154] The service transport module provides a data transmission channel for the interaction between the service interface module and the service management module. For example, the service transport module includes the Android Interface Definition Language (AIDL) interface layer and the binder driver layer. The AIDL interface layer can be used to support the binder driver layer in implementing data transmission.

[0155] The service management module is used to manage the various services provided by the service function modules, such as... Figure 5 As shown, the service management module may include a service identification module, a service registration module, a service distribution module, and a service deregistration module. The service identification module is used to determine which service function module to interact with the lower layer through based on the instructions from the upper layer. The service distribution module is used to distribute tasks to the service function modules determined by the service identification module. The service registration module is used to register callbacks, which are used to feed back the processing results of the chip's underlying algorithm on the video material to the upper layer. The service deregistration module is used to deregister callbacks.

[0156] The service function module is used to interact with the HAL layer. It includes a HAL service listening module, a function distribution module, and a state management module. The HAL service listening module listens for and reports the processing results fed back by the HAL layer. The function distribution module distributes data to the algorithms in the HAL layer to achieve the interface with the algorithms in the HAL layer. The state management module feeds back the state of the HAL layer to the upper layer. For example, it feeds back the algorithm initialization status to the upper layer when the algorithm is initialized, that is, it feeds back whether the algorithm has been initialized successfully.

[0157] The Hardware Abstraction Layer (HAL) encapsulates Linux kernel drivers, providing interfaces to higher-level systems. It hides the hardware interface details of specific platforms, providing the operating system with a virtual hardware platform that is hardware-independent and portable across multiple platforms. For example, the HAL provides an algorithm implementation layer, including modules for highlight fragment algorithms, resolution reduction algorithms, face detection algorithms, video acceleration, and image super-resolution algorithms. The highlight fragment algorithm module provides an image analysis algorithm (and highlight fragment algorithm) for the image chip. This algorithm scores each frame based on its color, texture features, quality, frame interpolation with preceding and following frames, and edge change rate. The score serves as the basis for evaluating whether a frame is a highlight fragment. For instance, a frame with a score greater than or equal to 60 is considered a highlight fragment. Highlight fragment algorithms typically include multiple analysis strategies, allowing different strategies to be used for different video materials. The resolution reduction algorithm module reduces image resolution, converting high-resolution images into low-resolution images. The face detection algorithm module is used to detect face regions from images. The video acceleration module uses hardware to replace software algorithms, reducing the load on the central processing unit (CPU) during video decoding. The image super-resolution algorithm module improves image resolution by converting low-resolution images into high-resolution images. In implementation, each algorithm module can provide different functional interfaces, allowing the service function layer to call the corresponding interfaces according to business needs. For example, calling the algorithm module's initialization interface allows the algorithm module to initialize itself, and calling the algorithm module's material analysis interface allows the algorithm module to perform material analysis.

[0158] It should be noted that, Figure 5The layers in the illustrated software architecture and the components contained within each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more layers than illustrated, such as a system library (FWK LIB) layer and a kernel layer. Furthermore, each layer may include more or fewer components than illustrated, which is not limited in this application.

[0159] It is understood that, in order to implement the video segment processing method in the embodiments of this application, the electronic device includes hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments.

[0160] In the above Figure 5 Based on the illustrated embodiments, the methods provided in this application will be described below. Please refer to... Figure 6 , Figure 6 This is a flowchart illustrating a method for invoking an algorithm according to an exemplary embodiment. As an example and not a limitation, the method can be derived from the above. Figure 5 The electronic device shown performs the operation, and the next step assumes that the electronic device is a mobile phone and the mobile phone is connected via... Figure 5 The following example illustrates the interaction between multiple modules. This method may include some or all of the following:

[0161] Step 6001: The video editing business layer receives the user's operation to enable the one-click video creation function.

[0162] For example, please refer to Figure (c) in 1. When a user opens the gallery and triggers the display of the gallery interface A1 on the mobile phone, when the user clicks the "One-Click Movie" control in the gallery, the mobile phone receives the operation to enable the one-click movie function through the video editing service layer.

[0163] Step 6002: The video editing business layer loads and displays the materials in the gallery.

[0164] As mentioned earlier, the materials in the gallery may include image materials and / or video materials. In one example, after receiving a user's request to enable the one-click photo creation function, the phone displays the gallery interface A3 through the video editing service layer. Gallery interface A3 includes all image materials and / or video materials in the gallery. For another example, please refer to... Figure 1In Figure (d), after receiving the user's operation to enable the one-click video creation function, the mobile phone displays the gallery interface A2 through the video editing service layer. Gallery interface A2 includes recently captured materials. The user can select any material in gallery interface A2. In response to the user's selection operation of any material in gallery interface A2, the mobile phone displays gallery interface A3 through the video editing service layer.

[0165] Step 6003: The video editing business layer receives the user's selection of target materials and confirms the execution of the one-click video creation operation. The target materials include multiple target video materials and multiple target image materials.

[0166] After the video editing business layer loads and displays the materials in the gallery, the user can select the materials they want to use from the gallery according to their needs. This embodiment illustrates this using the example of a user selecting multiple image materials and multiple video materials. After the user finishes making their selections, they trigger a one-click video creation operation, for example... Figure 1 As shown in Figure (f) in the middle, or as... Figure 2 As shown in Figure (a), when a user clicks the checkmark control in A3 of the gallery interface, the video editing business layer receives confirmation from the user to execute the one-click video creation operation.

[0167] Step 6004: The video editing business layer calls the initialization interface of the solution interface layer through the application function layer.

[0168] After receiving confirmation from the user to perform the one-click video creation operation, in order to use the chip's underlying algorithm to process the selected target material, the video editing business layer calls the initialization interface of the solution interface layer so that the chip's underlying algorithm can be initialized through the solution interface layer.

[0169] It should be noted that the media middleware framework layer may include multiple solution interface layers. Different solution interface layers can be used to connect to different services. It is easy to understand that the solution interface layer in this embodiment is used to connect to the one-click video production service.

[0170] Step 6005: The solution interface layer sends initialization parameters to the pipeline interface layer. The initialization parameters include algorithm name A, algorithm name B, and interface indication information tp1 for the initialization interface.

[0171] Algorithm A is the name of the resolution reduction algorithm. Algorithm B is the name of the highlight fragment algorithm.

[0172] Interface indication information is used to indicate the type of interface to be called. For example, the interface indication information for the initialization interface is used to indicate that the interface to be called is the initialization interface.

[0173] As mentioned earlier, the solution interface layer is used to interface with the one-click video creation service. Therefore, when the solution interface layer receives a call from the video editing service layer to its initialization interface for the one-click video creation service, it can determine which algorithms need to be initialized and then send initialization parameters to the pipeline interface layer. The initialization parameters include the algorithm name of the algorithm to be initialized, as well as the interface indication information of the interface in the algorithm service framework layer (i.e., the initialization interface) that needs to be called to initialize this algorithm.

[0174] Step 6006: The pipeline interface layer calls the initialization interface L1 of the service interface module 1 corresponding to the algorithm name A, and sends the initialization instruction r1 to L1, with r1 carrying the identifier a.

[0175] The identifier 'a' uniquely identifies the instruction associated with the current service request for the one-click video creation function. This instruction includes those transmitted between modules from the moment the user confirms the one-click video creation operation until the video clip is generated. Identifier 'a' can be assigned by the pipeline interface layer. That is, after receiving the initialization parameters from the solution interface layer, the pipeline interface layer assigns a unique identifier 'a' to this service request for the one-click video creation function, and this identifier 'a' is carried throughout subsequent module interactions. In one example, identifier 'a' can be randomly generated.

[0176] As mentioned above, the service interface layer includes at least one service interface module, and each service interface module corresponds to an algorithm. For an example of this application, please refer to... Figure 7 The service interface layer includes multiple service interface modules, such as service interface module 1 corresponding to the down-resolution algorithm, service interface module 2 corresponding to the highlight fragment algorithm, etc., and so on. Figure 5 As shown, each service interface module includes multiple interfaces. During the algorithm initialization phase, the pipeline interface layer calls the initialization interface in the service interface module corresponding to each algorithm that needs to be initialized, sending an initialization command to the initialization interface to request the algorithm initialization. For example, to initialize the resolution reduction algorithm, the pipeline interface layer calls the initialization interface L1 in service interface module 1 according to the algorithm name A and the interface indication information tp1, sending an initialization command r1 carrying the identifier a to the initialization interface L1.

[0177] Step 6007: Service interface module 1 sends an initialization command r2 to the service transport layer. r2 includes module identifier 1, interface indication information tp1, and identifier a.

[0178] Module Identifier 1 is used to uniquely identify Service Interface Module 1.

[0179] In order to initialize the resolution reduction algorithm at the chip level, the service interface module 1 receives the initialization instruction r1 through the initialization interface L1 and then sends the initialization instruction r2 to the service transport layer. The initialization instruction r2 carries the module identifier 1 of the service interface module 1 and the interface indication information tp1 of the initialization interface, so that the underlying module can know which service interface module sent r2 through which interface.

[0180] Step 6008: The service transport layer sends an initialization command r2 to the service management layer.

[0181] In one example, the service transport layer transmits the initialization command r2 to the service management layer via the binder driver layer. For example, the code implementation for transmitting the initialization command r2 via the binder driver layer is as follows: Figure 7 The code is shown in the image.

[0182] Step 6009: The service management layer creates a service execution instance T1 corresponding to service function module 1 based on module identifier 1.

[0183] As mentioned earlier, the service function layer includes at least one service function module, and each service function module interfaces with an algorithm. Please refer to [link / reference needed]. Figure 7 The service interface layer includes service function module 1 corresponding to the down-resolution algorithm and service function module 2 corresponding to the highlight fragment algorithm. Thus, during algorithm initialization, the service management layer can initialize the down-resolution algorithm through service function module 1. In implementation, since service function module 1 is virtualized, it typically needs to be instantiated to interact with the underlying layer; that is, a corresponding service execution instance is created based on the template of service function module 1. In this embodiment, to serve different services in parallel, the service management layer can create different service execution instances for each service based on the template of service function module 1. Each service execution instance has the same function but different private data areas, such as private caches and callback listeners. Thus, for different services, the service management layer can use the underlying down-resolution algorithm in parallel through different service execution instances. Here, for the one-click image processing service, the service management layer creates a corresponding service execution instance T1 based on the template of service function module 1.

[0184] As an example of this application, the specific implementation of the service management layer creating a service execution instance T1 based on module identifier 1 may include: the service identification module receiving an initialization instruction r2 and parsing the initialization instruction r2. Based on the parsed module identifier 1, the service identification module determines that the algorithm to be interfaced is a resolution reduction algorithm, and instructs the service distribution module to interact with the resolution reduction algorithm module through service function module 1. The service distribution module creates the service execution instance T1 based on service function module 1.

[0185] Step 6010: The service management layer sends an initialization command r2 to the service execution instance T1 of the service function layer, and manages the identifier a and the service execution instance T1.

[0186] In one example, the service management layer can call the service execution instance T1 through the service distribution module to send the initialization command r2 to the service execution instance T1.

[0187] Furthermore, since the one-click image processing service involves subsequent interactions with the resolution downscaling algorithm module via service execution instance T1, the service management module can bind identifier 'a', module identifier '1', and service execution instance T1 during algorithm initialization. This allows the module to locate the corresponding service execution instance T1 based on identifier 'a', enabling interaction with the resolution downscaling algorithm module through service execution instance T1. Moreover, the service management module manages the lifecycle of identifier 'a' and service execution instance T1, ensuring that identifier 'a' and service execution instance T1 remain bound and exist throughout the one-click image processing service's duration.

[0188] Step 6011: Initialize the service execution instance T1 in the service function layer.

[0189] In one example, the service function layer initializes the cache and callback listener for the service execution instance T1, where the callback listener is used to listen for the processing results fed back by the HAL layer's downscaling algorithm module.

[0190] Of course, this embodiment is illustrated using the example that service execution instance T1 has not been created in advance. In another embodiment, if service execution instance T1 has been created in advance, that is, if the corresponding service execution instance T1 can be found according to the identifier a, it means that the resolution reduction algorithm has been initialized. In this case, the service management layer can return an initialization success notification, that is, it is not necessary to execute the operations of steps 6009 to 6014.

[0191] Step 6012: The service function layer sends an initialization command r2 to the resolution reduction algorithm module through the service execution instance T1 according to tp1.

[0192] In one example, after the service execution instance T1 receives the initialization instruction r2 through the function distribution module, it determines that initialization is required based on the interface indication information tp1 in the initialization instruction r2. At this time, it calls the initialization interface in the resolution reduction algorithm module, thereby enabling the resolution reduction algorithm module to perform the resolution reduction algorithm initialization.

[0193] Step 6013: In response to the initialization instruction r2, the resolution reduction algorithm module performs algorithm initialization.

[0194] In response to the initialization instruction r2, the down-resolution algorithm module starts the down-resolution algorithm, thereby performing algorithm initialization.

[0195] In one example, the down-resolution algorithm module determines whether the down-resolution algorithm has been initialized based on the identifier 'a'. If it has not been initialized, the algorithm is initialized; otherwise, if it has been initialized, the algorithm is not initialized again.

[0196] Step 6014: The resolution downscaling algorithm module returns an initialization success notification a.

[0197] The initialization success notification 'a' indicates that the down-resolution algorithm has been successfully initialized.

[0198] In this embodiment, the resolution reduction algorithm module sends an initialization success notification a to the service execution instance T1 in the service function layer based on the identifier a. The resolution reduction algorithm module includes multiple algorithm execution instances, each corresponding to a service execution instance in the service function layer. Thus, the resolution reduction algorithm module can determine the corresponding algorithm execution instance based on the identifier a, and send the initialization success notification a to the service execution instance T1 in the service function layer through the determined algorithm execution instance.

[0199] In one instance, service execution instance T1 receives initialization success notification 'a' through the state management module. Then, service execution instance T1 sends initialization success notification 'a' to the service management layer, which in turn sends it to the service transport layer. The service transport layer sends initialization success notification 'a' to service interface module 1, which receives it through initialization interface L1. Service interface module 1 then sends initialization success notification 'a' to the pipeline interface layer. If service interface module 1 sends initialization success notification 'a' to the pipeline interface layer through initialization interface L1, the pipeline interface layer can then determine that the resolution reduction algorithm has been successfully initialized.

[0200] Step 6015: The pipeline interface layer calls the initialization interface L2 of the service interface module 2 corresponding to the algorithm name B, and sends the initialization instruction r3 to L2, with r3 carrying the identifier a.

[0201] As mentioned earlier, the service interface layer includes service interface module 2 corresponding to the highlight fragment algorithm. During the algorithm initialization phase, the pipeline interface layer calls the initialization interface L2 in service interface module 2 based on the algorithm name B and the interface indication information ty1, and sends the initialization instruction r3 to the initialization interface L2 in service interface module 2. As mentioned earlier, the same identifier is used for the same execution task of the one-click image processing service, therefore the initialization instruction r3 carries the identifier a.

[0202] It should be noted that there is no strict order of execution between steps 6006 and 6015. As an example of this application, steps 6006 and 6015 can be executed in parallel, that is, the pipeline interface layer can call the initialization interface L1 and initialization interface L2 of the service interface layer respectively to initialize the down-resolution algorithm and the specular fragment algorithm in parallel.

[0203] Step 6016: Service interface module 2 sends an initialization command r3 to the service transport layer through initialization interface L2. r3 includes module identifier 2, tp1 and identifier a.

[0204] In order to initialize the high-light fragment algorithm at the chip's underlying level, service interface module 2 receives initialization instruction r3 through initialization interface L2 and then sends initialization instruction r3 to the service transport layer. To help the underlying modules know which interface of which service interface module is being called, the initialization instruction r3 carries, in addition to the identifier a, the module identifier 2 of service interface module 2, and the interface indication information tp1 of initialization interface L2.

[0205] Step 6017: The service transport layer sends an initialization command r3 to the service management layer.

[0206] In one example, the service transport layer transmits the initialization instruction r3 to the service management layer through the binder driver layer.

[0207] Step 6018: The service management layer creates a service execution instance T2 corresponding to service function module 2 based on module identifier 2.

[0208] Since Service Function Module 2 is the service function module corresponding to the highlight fragment algorithm, the service management layer can initialize the highlight fragment algorithm through Service Function Module 2 during the algorithm initialization process. Similarly, Service Function Module 2 is virtualized, so in order to interact with the underlying layer, Service Function Module 2 needs to be instantiated, that is, the corresponding service execution instance 2 is created based on the template of Service Function Module 2.

[0209] As an example of this application, the specific implementation of the service management layer creating a service execution instance T2 based on module identifier 2 may include: the service identification module receiving an initialization command r3 and parsing the initialization command r3; the service identification module determining, based on the parsed module identifier 2, that the algorithm to be interfaced is the highlight fragment algorithm, and instructing the service distribution module to interact with the highlight fragment algorithm module through service function module 2; and the service distribution module creating the service execution instance T2 based on service function module 2.

[0210] Step 6019: The service management layer sends an initialization command r3 to the service execution instance T2 of the service function layer, and manages the identifier a and the service execution instance T2.

[0211] In one example, the service management layer can call the service execution instance T2 through the service distribution module to send the initialization command r3 to the service execution instance T2.

[0212] Furthermore, the service management module binds identifier 'a', service identifier 2, and service execution instance T2, making it easier to find the corresponding service execution instance T2 based on identifier 'a', thus enabling interaction with the highlight fragment algorithm module through service execution instance T2. Additionally, the service management layer manages the lifecycle of identifier 'a' and service execution instance T2.

[0213] Step 6020: Initialize the service execution instance T2 in the service function layer.

[0214] In one example, the service function layer initializes the cache and callback listener for the service execution instance T2, where the callback listener is used to listen for the processing results fed back by the highlight fragment algorithm module of the HAL layer.

[0215] Of course, this embodiment is illustrated using the example that service execution instance T2 has not been created in advance. In another embodiment, if service execution instance T2 has been created in advance, that is, if the corresponding service execution instance T2 can be found according to the identifier a, it means that the highlight fragment algorithm has been initialized. In this case, the service management layer can return an initialization success notification, that is, it is not necessary to execute the operations of steps 6018 to 6023.

[0216] Step 6021: The service function layer sends the initialization command r3 to the highlight fragment algorithm module through the service execution instance T2 according to tp1.

[0217] Step 6022: In response to the initialization command r3, the highlight fragment algorithm module performs algorithm initialization.

[0218] In response to the initialization command r3, the highlight fragment algorithm module starts the highlight fragment algorithm to perform algorithm initialization.

[0219] Step 6023: The highlight fragment algorithm module returns an initialization success notification b.

[0220] The initialization success notification b is used to indicate that the specular fragment algorithm has been successfully initialized.

[0221] In this embodiment, the highlight fragment algorithm module sends an initialization success notification b to the service execution instance T2 in the service function layer based on the identifier 'a'. Service execution instance T2 receives the initialization success notification b through the state management module. Then, service execution instance T2 sends the initialization success notification b to the service management layer, which in turn sends it to the service transport layer. The service transport layer sends the initialization success notification b to the service interface module 2, which receives it through the initialization interface L2. Service interface module 2 then sends the initialization success notification b to the pipeline interface layer. If service interface module 2 sends the initialization success notification b to the pipeline interface layer through the initialization interface L2, the pipeline interface layer can determine that the highlight fragment algorithm initialization was successful.

[0222] It should be noted that the embodiments in this application only illustrate the algorithms used in the one-click image processing service, including the resolution reduction algorithm and the highlight fragment algorithm. In another example, if other algorithms are used, they can all be initialized according to a similar process described above, which will not be described in detail here.

[0223] Step 6024: The pipeline interface layer calls the performance analysis interface L3 of service interface module 2 and sends a speed query command r4 to the performance analysis interface L3, with the identifier a carried in r4.

[0224] The speed query command `r4` is used to query the chip analysis speed of the highlight clip algorithm. Chip analysis speed indicates the analysis capability of the highlight clip algorithm. Typically, the chip analysis speed is a multiple, such as 3 or 5. For example, a chip analysis speed of 3 means that a video clip of duration `n` requires n / 3 of the analysis time (e.g., 10 seconds for a 30-second video clip). Similarly, a chip analysis speed of 5 means that a video clip of duration `n` requires n / 5 of the analysis time (e.g., 6 seconds for a 30-second video clip). Chip analysis speed is usually related to the phone's chip model.

[0225] To facilitate the allocation of reasonable analysis time for each target video clip among multiple target video clips, after the highlight clip algorithm initialization is complete, the pipeline interface layer calls the performance analysis interface L3 of service interface module 2 in the service interface layer to request a query for the chip analysis speed of the highlight clip algorithm. To allow the underlying modules to know which service execution instance is interacting with the highlight clip algorithm module, the speed query command carries the identifier 'a'.

[0226] Step 6025: Service interface module 2 sends a speed query command r5 to the service transport layer through the performance analysis interface L3. r5 includes identifier a, module identifier 2 and interface indication information tp2.

[0227] The interface indication information tp2 is used to indicate the performance analysis interface.

[0228] Step 6026: The service transport layer sends a speed query command r5 to the service management layer.

[0229] Step 6027: The service management layer sends a speed query command r5 to the service execution instance T2 of the service function layer based on the identifier a and the module identifier 2.

[0230] Since the service management layer binds identifier a, module identifier 2, and service execution instance T2 in the algorithm initialization interface service management layer, after receiving the speed query command r5, the service management layer can determine that it needs to interface with the highlight fragment algorithm through service function module 2 based on module identifier 2. Then, it can find the service execution instance T2 corresponding to service function module 2 based on identifier a and send the speed query command r5 to service execution instance T2.

[0231] Step 6028: The service function layer calls the chip analysis speed interface of the highlight fragment algorithm module through service execution instance 2 based on tp2.

[0232] Based on the interface instruction information tp2, it can be determined that the chip analysis speed needs to be queried. Therefore, service execution instance 2 calls the chip analysis speed interface of the highlight fragment algorithm module to request a query on the chip analysis speed of the highlight fragment algorithm.

[0233] Step 6029: The highlight fragment algorithm module sends the chip analysis speed to the service function layer.

[0234] As an example of this application, the highlight fragment algorithm module sends the chip analysis speed to the service execution instance 2 of the service function layer according to the identifier a.

[0235] Step 6030: The service function layer returns the chip analysis speed.

[0236] In implementation, the service function layer receives the chip analysis speed through service execution instance T2 and sends it to the service management layer. The service management layer sends the chip analysis speed to the service transport layer, which in turn sends it to service interface module 2 in the service interface layer. Service interface module 2 receives the chip analysis speed through the performance analysis interface and sends it to the pipeline interface layer.

[0237] Step 6031: The pipeline interface layer sends the chip analysis speed to the solution interface layer.

[0238] Step 6032: Analysis speed of the solution interface layer memory chip.

[0239] In one example, the solution interface layer receives the chip analysis speed through the analysis performance query interface and then stores the chip analysis speed.

[0240] Step 6033: The solution interface layer sends an initialization completion notification to the video editing business layer.

[0241] After all algorithms have been successfully initialized and the chip analysis speed has been obtained, the solution interface layer sends an initialization completion notification to the video editing business layer. In this way, the video editing business layer can know that the initialization of the required algorithms has been completed.

[0242] Step 6034: The video editing business layer sends a video processing speed query command r6 to the solution interface layer. The video processing speed query command r6 carries the file descriptor fd1 of the target video material to be queried.

[0243] The video processing speed query command r6 is used to query the analysis speed of the target video material.

[0244] The file descriptor (fd) of a media file indicates its storage address; therefore, the file descriptor can also be called address indication information. For example, a file descriptor can be represented as fd = openfile(filepath), where filepath is the address information of the media file's storage address. For instance, the address information of the target video media v1 is represented as: filepath = / sdcard / DCIM / video.1.

[0245] As an example of this application, when multiple target video clips exist, the video editing business layer can query the analysis speed of each target video clip in the multiple target video clips one by one. That is, each time the video processing speed instruction carries the file descriptor of one of the multiple target video clips, for example, carrying the file descriptor fd1 of target video clip v1, to request the solution interface layer to query the analysis speed of target video clip v1. In implementation, the video editing business layer calls the analysis performance query interface of the solution interface layer to determine the analysis speed of the target video clip v1 to be queried through the analysis performance query interface.

[0246] Step 6035: The solution interface layer obtains the resolution and frame rate of the target video material based on fd1.

[0247] In one example, since fd1 is used to indicate the storage address of the target video clip to be queried, the solution interface layer can use the performance query interface to find the target video clip based on fd1, thereby obtaining the resolution and frame rate of the video frame of the target video clip.

[0248] Step 6036: The solution interface layer determines the analysis speed of the target video material based on the chip analysis speed, resolution, and frame rate.

[0249] The analysis speed of the first target video clip can be denoted as S1, the second as S2, and so on, with the analysis speed of the nth target video clip denoted as Sn. The analysis speed of a single target video clip is related to its resolution and frame rate. For example, the higher the resolution of the target video clip, the lower its analysis speed. The higher the frame rate of the target video clip, the lower its analysis speed. Since the resolution and frame rate of each target video clip are different, the analysis speed of each target video clip may be different.

[0250] In this embodiment, the analysis speed of a single target video clip can be determined based on the chip analysis speed and the video decoding speed. The video decoding speed can be calculated based on the resolution and frame rate. The process of calculating the analysis speed Sn of each target video clip is described in detail below.

[0251] In this embodiment of the application, the analysis speed Sn of a single target video clip can be calculated using the following formula (1):

[0252] S n =min(V1, V2)×α (1)

[0253] Where V1 represents the video decoding speed (decodeSpeed), V2 represents the chip analysis speed (chipestSpeed), and α represents the tolerance factor, with α set to 0.9 as an example. min(V1, V2) means taking the smaller value between V1 and V2.

[0254] In this embodiment, the chip analysis speed V2 is a performance parameter obtained from the HAL layer during initialization. For example, the chip analysis speed V2 can be 3x, 4x, or 5x. For ease of explanation, this embodiment uses a chip analysis speed of 5x as an example.

[0255] In this embodiment, the video decoding speed V1 can be calculated based on the resolution and frame rate of the target video material, on the basis of the default video decoding speed (denoted as Vd). The process of calculating the video decoding speed V1 is described in detail below. First, the video decoding speed V1 can be calculated based on the resolution and frame rate of the target video material using the following formula (2):

[0256] V1 = V d ×(R d / R A )×(F d / F A (2)

[0257] Where Vd represents the default video decoding speed, Rd represents the default video resolution, RA represents the actual resolution of the target video footage, Fd represents the default video frame rate, and FA represents the actual frame rate of the target video footage.

[0258] For example, the default video decoding speed Vd can be 4x, the default video resolution Rd can be 1080p (i.e., 1920×1080), and the default video frame rate Fd can be 30 frames per second.

[0259] Taking a video with an actual resolution RA of 720p (i.e., 1280×720) and an actual frame rate FA of 60 frames / second as an example, the video decoding speed V1 is calculated by substituting the above parameters into formula (2):

[0260] V1 = V d ×(R d / R A )×(R d / R A = 4 × (1920 × 1080) / (1280 × 720) × (30 / 60) = 4.5

[0261] Therefore, the video decoding speed V1 can be calculated using the above formula (2).

[0262] Next, we will calculate the analysis speed Sn of a single target video clip using the above formula (1). Here, we assume that the chip analysis speed V2 is 5 times the speed. We will then substitute the video decoding speed V1 and the chip analysis speed V2 into formula (1) to calculate the analysis speed S of a single target video clip. n =min(4.5,5)×0.9=4.5×0.9≈4.

[0263] Step 6037: The solution interface layer feeds back the analysis speed to the video editing business layer.

[0264] Step 6038: If there are unqueried target video materials, the video editing business layer continues to query the analysis speed of the next target video material.

[0265] As mentioned earlier, when multiple target video clips exist, the video editing business layer queries the analysis speed of each target video clip in turn. Therefore, after the solution interface layer returns the analysis speed of the target video clip, it determines whether there are any unqueried target video clips. If so, it continues to query the analysis speed of the next target video clip, that is, it returns to step 6034; otherwise, it proceeds to step 6039.

[0266] Step 6039: If there are no unqueried target video materials, the video editing business layer determines the total target analysis time based on the analysis speed of all target video materials and the preset analysis time of all target image materials.

[0267] The preset analysis duration can be set according to actual needs. That is, a preset analysis duration, such as 1 second, can be assigned to each target image material. Then, the total target analysis duration is determined based on the analysis speed of each target video material and the preset analysis duration of each target image material in all target video materials.

[0268] As an example of this application, in step 6039, the total target analysis time T can be calculated according to the following formulas (3)-(4) A :

[0269] T A =∑T Vn +∑T P (3)

[0270] T Vn =min(L n / S n ,β)(4)

[0271] Among them, T VnT represents the expected analysis duration of a single target video clip out of multiple target video clips. P This represents the preset analysis time for a single target image among multiple target image materials. The following explanation uses a preset analysis time of 1 second as an example. `min(Ln / Sn, β)` represents taking the minimum value between `Ln / Sn` and `β`. n Let β be the video duration of the nth target video clip. β is the upper limit of the expected analysis duration of a single target video clip. This ensures that the expected analysis duration of a single target video clip is within the limit range. In implementation, the upper limit of the expected analysis duration of a single target video clip can be set according to actual usage requirements, and this application embodiment does not impose any limitation.

[0272] Assuming the user selects n target video clips and m target image clips as target materials, the sum of the expected analysis times for these m target image clips is m. Accordingly, the total target analysis time T can be calculated using the following formula (5). A :

[0273] T A =∑min(L n / S n ,β)+m(5)

[0274] Therefore, when the user selects multiple target video materials and multiple target image materials as target materials, the total target analysis time T can be obtained by adding the sum of the expected analysis times of all videos to the sum of the preset analysis times of all target image materials. A .

[0275] It should be noted that the embodiments of this application are illustrated using the example of the user selecting target materials including multiple target video materials and multiple target image materials. In another example, if the user selects target materials including only multiple target video materials, the total target analysis time T is calculated according to the following formulas (6) and (7). A :

[0276] T A =∑T Vn (6)

[0277] T Vn =min(L n / S n ,β)(7)

[0278] Determine the total analysis time T for the target. A Then, the selected target material can be processed through the chip's underlying algorithms. For details, please refer to... Figure 8 The process includes the following steps:

[0279] Step 6040: The video editing business layer sends the material analysis command r7 to the solution interface layer. The material analysis command r7 carries the total target analysis duration and the file descriptor (fd) of each target material.

[0280] In implementation, the video editing business layer calls the highlight clip analysis interface and sends the material analysis command r7 to the highlight clip analysis interface. The highlight clip analysis interface calls the policy monitoring interface and sends the material analysis command r7 to the policy monitoring interface, so as to request the underlying layer to perform highlight clip analysis on each target material through the policy monitoring interface.

[0281] Step 6041: In response to r7, the solution interface layer sends an image analysis command r8 to the pipeline interface layer, r8 carrying the fd2 of the target image material to be analyzed.

[0282] As an example of this application, when the target materials include multiple target video materials and multiple target image materials, the multiple target image materials can be analyzed one by one first, and then the multiple target video materials can be analyzed one by one. Therefore, when the solution interface layer receives the material analysis instruction r7, it can send the image analysis instruction r8 to the pipeline interface layer, indicating that the target image materials should be analyzed first.

[0283] Optionally, if the highlight fragment algorithm module needs to obtain the material parameters (such as frame width, frame height, etc.) of the material to be analyzed in advance when performing highlight fragment analysis, the solution interface layer, after receiving the material analysis instruction and before step 6041, can also obtain the relevant parameters based on the file descriptors (fd) of each target material, and then instruct the pipeline interface layer to send them down. Correspondingly, the pipeline interface layer can call the parameter management interface in service interface module 2 to send the relevant parameters to the service transport layer, which in turn sends the relevant parameters to the service management layer. The service management layer then calls service execution instance T2 to send the relevant parameters to the highlight fragment algorithm module.

[0284] Optionally, while the parameter management interface sends relevant parameters to the underlying layer, it can also send a callback registration instruction to the service transport layer. The service transport layer then sends a service registration instruction to the service management layer. The service management layer receives the service registration instruction through the service identification module and then instructs the service registration module to perform callback registration. This callback is used to feed back the highlight segment analysis results of the video material to the upper layer.

[0285] Step 6042: The pipeline interface layer obtains the target image material based on fd2.

[0286] The pipeline interface layer retrieves the target image material to be analyzed from the corresponding storage address based on fd2.

[0287] Step 6043: The pipeline interface layer stores the target image material into the shared cache.

[0288] In one example, the shared cache is a cache used by both the media middleware framework layer and the algorithm implementation layer.

[0289] Step 6044: The pipeline interface layer sends an image compression instruction r8 to the down-resolution interface of service interface module 1. r8 carries the frame data address d1 and the identifier a. d1 is used to indicate the storage location of the target image material to be analyzed in the shared cache.

[0290] As an example of this application, before analyzing the target image material, the pipeline interface layer can first perform resolution reduction processing on the target image material using an underlying resolution reduction algorithm to improve the efficiency of subsequent analysis. Furthermore, since service interface module 1 in the service interface layer corresponds to the resolution reduction algorithm, the pipeline interface layer sends an image compression command to service interface module 1. In one example, the pipeline interface layer calls the resolution reduction interface of service interface module 1 and sends an image compression command to the resolution reduction interface to request resolution reduction processing of the target image material.

[0291] Optionally, the image compression instruction r8 can also carry the resolution of the target image material.

[0292] Step 6045: In response to r8, service interface module 1 sends an image compression instruction r9 to the service transport layer. r9 includes the identifiers a, d1, module identifier 1, and interface indication information tp3 for the down-resolution interface.

[0293] In one example, if the image compression instruction r8 also carries the resolution, the service interface module 1 can determine whether the target image material needs to be down-processed based on the resolution. For instance, if the resolution is less than a resolution threshold, it is determined that no down-processing is needed; if the resolution is greater than or equal to the resolution threshold, it is determined that down-processing is needed. If down-processing is determined to be needed, the service interface module 1 sends the image compression instruction r9 to the service transport layer through the down-processing interface. Otherwise, if down-processing is determined not to be needed, the service interface module 1 can send a notification to the pipeline interface layer indicating that down-processing is not required. The resolution threshold can be set according to actual needs, for example, a resolution threshold of 480p.

[0294] Step 6046: The service transport layer sends an image compression command r9 to the service management layer.

[0295] Step 6047: The service management layer determines the service execution instance T1 based on identifier a and module identifier 1.

[0296] After receiving the image compression command r9, the service management layer parses the command to obtain the identifier 'a' and module identifier '1'. As mentioned earlier, during the algorithm initialization phase, the service management layer binds the identifier 'a', module identifier '1', and the service execution instance T1 corresponding to service function module 1. Therefore, the service management layer can find the corresponding service execution instance T1 based on the identifier 'a' and module identifier '1'.

[0297] Step 6048: The service management layer sends an image compression command r9 to the service execution instance T1 of the service function layer.

[0298] Since the service function layer interacts with the resolution reduction algorithm module through the service execution instance T1 for the one-click image generation service, the service management layer calls the service execution instance T1 of the service function layer and sends the image compression command r9 to the service execution instance T1 to request the resolution reduction algorithm module to perform resolution reduction processing through the service execution instance T1.

[0299] Step 6049: The service function layer sends the image compression instruction r9 to the down-resolution algorithm module through the service execution instance T1 according to tp3.

[0300] Service execution instance T1 determines the request for resolution downscaling based on tp3, and therefore calls the resolution downscaling interface in the resolution downscaling algorithm module, sending the image compression command r9 to the resolution downscaling interface to request resolution downscaling. Afterwards, service execution instance T1 listens for feedback from the resolution downscaling algorithm module through a callback listener.

[0301] Step 6050: In response to the image compression instruction r9, the resolution reduction algorithm module performs resolution reduction processing on the target image material based on d1.

[0302] In one example, the down-resolution algorithm module reads the target image material from the shared buffer based on frame data address d1. Typically, the read target image material is in NV12 format. Since down-resolution processing supports bitmap format, the target image material is decoded into bitmap format, and then down-resolution is applied to the bitmap format target image material, for example, reducing it to 480p image data. The 480p image data is then converted back to NV12 format to obtain the low-resolution target image material.

[0303] Step 6051: The resolution reduction algorithm module stores the low-resolution target image material into the shared cache.

[0304] Step 6052: The resolution reduction algorithm module sends frame data address d2 to the service execution instance T1 of the service function layer. Frame data address d2 is used to indicate the storage location of the low-resolution target image material in the shared cache.

[0305] The resolution reduction algorithm module sends frame data address d2 to the service execution instance T1 of the service function layer according to the identifier a.

[0306] It is worth mentioning that after the down-resolution processing, the down-resolution algorithm module stores the low-resolution target image material in a shared cache and returns the frame data address d2, which can reduce the amount of data transmission.

[0307] Step 6053: The service function layer returns the frame data address d2 through the service execution instance T1.

[0308] In one example, the service function layer listens for feedback from the down-resolution algorithm module through a callback listener in service execution instance T1. When it receives frame data address d2, it sends frame data address d2 to the service management layer through service execution instance T1. The service management layer then sends frame data address d2 to the service transport layer, for example, by sending frame data address d2 to the service transport layer through the service distribution module. The service transport layer sends frame data address d2 to service interface module 1. Service interface module 1 then sends frame data address d2 to the pipeline interface layer, for example, by receiving frame data address d2 sent by the service transport layer through the down-resolution interface and then sending frame data address d2 to the pipeline interface layer.

[0309] In one example, after the pipeline interface layer receives frame data address d2, it can delete the target image material before processing corresponding to this frame data address d2 in the shared cache to save cache space.

[0310] Step 6054: The pipeline interface layer sends an image analysis command r10 to the image analysis interface of the service interface module 2. The image analysis command r10 carries the frame data address d2 and the identifier a.

[0311] After down-resolution processing of the target image material, the underlying layer is requested to perform highlight segment analysis based on the down-resolution target image material. To this end, the pipeline interface layer sends image analysis command r10 to the image analysis interface of service interface module 2.

[0312] Step 6055: In response to r10, service interface module 2 sends an image analysis instruction r11 to the service transport layer. r11 carries d2, identifier a, module identifier 2 and interface indication information tp4 of the image analysis interface.

[0313] Step 6056: The service transport layer sends an image analysis command r11 to the service management layer.

[0314] Step 6057: The service management layer determines the service execution instance T2 based on identifier a and service identifier 2.

[0315] Since the service management module binds identifier a, service identifier 2 and service execution instance T2 corresponding to service function module 2 during the algorithm initialization phase, the service management module can find the corresponding service execution instance T2 based on identifier a and service identifier 2.

[0316] Step 6058: The service management layer sends an image analysis command r11 to the service execution instance T2 of the service function layer.

[0317] Since the service function module 2 interacts with the highlight fragment algorithm module through the service execution instance T2 for the one-click image generation service, the service management module calls the service execution instance T2 and sends an image compression command to the service execution instance T2 so that the service execution instance T2 can send a command to the highlight fragment algorithm module to perform highlight fragment analysis.

[0318] Step 6059: The service function layer sends the image analysis instruction r11 to the highlight fragment algorithm module through the service execution instance T2 according to tp4.

[0319] Based on tp4, service execution instance T2 can determine that image analysis processing needs to be requested. At this time, service execution instance T2 can call the image analysis interface of the highlight fragment algorithm module and send the image analysis command r11 to the image analysis interface. After that, service execution instance T2 listens for the feedback from the highlight fragment algorithm module through a callback listener.

[0320] Step 6060: In response to the image analysis instruction r11, the highlight fragment algorithm module determines the score value of the target image material based on the frame data address d2.

[0321] The highlight fragment algorithm module reads low-resolution target image material from shared memory according to frame data address d2 in the image analysis instruction, and then analyzes the low-resolution target image material through the highlight fragment algorithm to determine the score value of the low-resolution target image material.

[0322] Step 6061: The highlight fragment algorithm module sends the score value to the service execution instance T2 of the service function layer.

[0323] The highlight fragment algorithm module can determine who needs to provide the score value based on the identifier 'a', that is, to send the score value to the service execution instance T2 of the service function module 2 corresponding to identifier 'a'.

[0324] Step 6062: The service function layer returns the score value through the service execution instance T2.

[0325] In one example, the service function layer listens for feedback from the highlight fragment algorithm module through a callback listener in service execution instance T2. When a score is received, it sends the score to the service management layer through service execution instance T2. The service management layer then sends the score to the service transport layer, for example, through the service distribution module. The service transport layer sends the score to the image analysis interface of service interface module 2. Service interface module 2 then sends the score to the pipeline interface layer through the image analysis interface.

[0326] Step 6063: The pipeline interface layer sends the score value to the solution interface layer.

[0327] Step 6064: If there are unprocessed target image materials, the solution interface layer continues to process the target image materials of the next frame.

[0328] After receiving the score, the Pipeline interface layer checks if there are any unprocessed target image clips. If so, it continues with the down-resolution processing and highlight fragment analysis of the next frame's target image clip. Otherwise, if no such clips are found, it means the analysis of all target image clips has been completed. The next step is to analyze multiple target video clips one by one. For details, please refer to... Figure 9 The process includes the following steps.

[0329] Step 6065: If there are no unprocessed target image materials, the solution interface layer determines the difference between the total target analysis time and the total image analysis time to obtain the expected time.

[0330] The total image analysis time refers to the total time spent analyzing all target image materials. In the above process, during the resolution downscaling and highlight segment analysis of each target image material, the pipeline interface layer can calculate the total analysis time for all target image materials. After analyzing multiple target image materials, the next step is to analyze multiple target video materials. Since the total target analysis time was determined in the preceding process, after calculating the total image analysis time, the total target analysis time is subtracted from the total image analysis time to obtain the expected time for the remaining unanalyzed target image materials. In other words, it is expected that the analysis of all unanalyzed target video materials will be completed within this expected time.

[0331] It should be noted that the embodiments in this application are illustrated using the example of target image materials included in the target materials. In another example, if the target materials do not include target image materials, but only include multiple target video materials, then after the video editing business layer sends the material analysis instruction to the solution interface layer, the solution interface layer directly performs the following operations. Furthermore, it is easy to understand that in this case, the expected duration is the total target analysis duration.

[0332] Step 6066: The solution interface layer queries the video duration of each target video material based on the file descriptor (fd) of each target video material.

[0333] For any given target video clip, the pipeline interface layer reads the target video clip from its corresponding storage address based on its file descriptor (fd), thereby determining the video duration of that target video clip. This method can be used to determine the video duration of each target video clip.

[0334] Step 6067: The solution interface layer determines the analysis duration of each target video clip based on the expected duration and the video duration of each target video clip.

[0335] The analysis duration allocated to each target video clip is denoted as T. n For example, the analysis time allocated to the first target video clip is T1, the analysis time allocated to the second target video clip is T1, and so on, with the analysis time allocated to the nth target video clip being T. n .

[0336] As an example of this application, the specific implementation of determining the analysis duration of each target video material based on the expected duration and the video duration of each target video material may include: distributing a portion of the expected duration in a predetermined proportion equally to each target video material, and distributing the remaining portion of the expected duration to each target video material according to the weight proportion corresponding to the target video material.

[0337] The preset ratio can be set according to actual usage needs, and this embodiment does not limit it. For example, the preset ratio can be 50%. That is, the desired duration T... V 50% is evenly allocated to each target video clip, and then the desired duration T is calculated. V The remaining 50% is allocated to each target video clip according to a weighted ratio. The following explanation uses a preset ratio of 50% (i.e., 0.5) as an example.

[0338] Specifically, the basic analysis duration T allocated to a single target video clip can be calculated using the following formula (8). n1 :

[0339] T n1 =T V ×0.5 / n(8)

[0340] Where n represents the number of target video clips to be analyzed.

[0341] The weighted analysis duration T assigned to a single target video clip is calculated using the following formulas (9)-(11). n2 :

[0342] T n2 =T V ×0.5×(W n / W A (9)

[0343] W n =log2(1.5+L) n / L A ) / log2(1.5+S n / S A (10)

[0344] W A =W1 + ... + W n (11)

[0345] Among them, W n This represents the weight value of a single target video clip. For example, W1 represents the weight of the first target video clip, W2 represents the weight of the second target video clip, and so on. n L represents the weight of the nth target video clip. A This represents the sum of the video durations of all target source videos. S A This represents the sum of the analysis speeds of all target video footage. W A This represents the sum of the weight values ​​of all target video materials. It should be noted that 1.5 in formula (10) is a preset value for the algorithm, which can be set according to actual usage requirements. This application embodiment does not limit this setting.

[0346] Thus, the analysis time T allocated to a single target video clip n It consists of two parts, namely T n =T n1 +T n2 .

[0347] In this embodiment, the longer the target video material, the more analysis time should be allocated to it. In other words, the video length L of a certain target video material... n The sum of the video durations of all target video clips L A The proportion L n / LA The larger the value, the more analysis time is allocated to the target video material.

[0348] In this embodiment, the faster the analysis speed of a target video clip, the less analysis time should be allocated. In other words, the analysis speed S of a certain target video clip... n The sum of the analysis speeds of all target video footage, S A The ratio S n / S A The larger the value, the less analysis time is allocated to the target video material.

[0349] Step 6068: The solution interface layer determines the analysis strategy corresponding to each target video material based on the analysis duration of each target video material.

[0350] As an example of this application, the highlight fragment algorithm module can provide a variety of analysis strategies, such as full analysis strategy, sparse key fragment analysis strategy, dense key fragment analysis strategy, and instantaneous decoding refresh (IDR) frame sampling analysis strategy.

[0351] The analysis strategies include: Full-frame analysis: This involves including every frame of the video in the analysis, encompassing all highlight segments, but it is relatively time-consuming and resource-intensive. I-frame-based simple analysis: When the allocated analysis time is only sufficient for analyzing three images, this strategy analyzes three images located at different positions in the video, using the image with the highest score as the starting point for highlight segment results, and returning a video segment of a certain duration as the result. Dense key segment analysis: This strategy calculates the number and duration of the analyzed video segments, determines the reasonable interval between adjacent segments, and then evenly distributes the analyzed segments to cover highlight segments in different positions as much as possible. Sparse key segment analysis: This strategy calculates the same number and duration of video segments as the dense strategy, but differs in that it distributes the interval between adjacent segments unequally. This algorithm primarily analyzes segments in the early and middle sections of the video, with some distribution at the end.

[0352] Different analysis strategies can be used to analyze highlight segments for target video materials with different analysis durations. Therefore, for any target video material among all target video materials, after determining the analysis duration of this target video material, the solution interface layer determines the corresponding analysis strategy for this target video material based on the analysis duration of this target video material, so as to instruct the highlight analysis algorithm module to use which analysis strategy to perform highlight segment analysis on this target video material.

[0353] As an example of this application, the solution interface layer can determine the analysis strategy corresponding to each target video clip based on the analysis duration, video duration, and video analysis rate of each target video clip. The video analysis rate is a performance parameter of the electronic device itself and is usually fixed.

[0354] In one possible scenario, the solution interface layer can determine the percentage of video frames that can be processed within the analysis time allocated to each target video clip, based on the video duration, analysis duration, and video analysis rate, and then determine the analysis strategy corresponding to each target video clip based on that percentage.

[0355] For example, the target video clip has a duration of 10 seconds, an analysis time of 1 second, and a video analysis rate of 5. Thus, the solution interface layer can calculate the analysis time required to analyze all the video frames of the target video clip based on its duration and analysis rate. For instance, the analysis time required to analyze all the video frames is the video duration divided by the analysis rate, which is 10 seconds / 5 = 2 seconds. The allocated analysis time for the target video clip is 1 second. This means that within the allocated analysis time, the percentage of the target video clip that can be analyzed is 1 second / 2 seconds = 50%.

[0356] Optionally, if the percentage of video frames in the target video clip that can be processed within the analysis time allocated to the target video clip is greater than a first preset threshold, then the analysis strategy allocated to the target video clip is a full-scale analysis strategy. For example, the first preset threshold is 100%.

[0357] For example, if the analysis time allocated to the target video material is 2.2 seconds, and the analysis time required to analyze all the video frames in the target video material is 2 seconds, then the percentage of video frames in the target video material that can be processed within the analysis time allocated to the target video material is 2.2 / 2 = 110%, which is greater than the first preset threshold of 100%. Therefore, the analysis strategy allocated to the target video material is the full analysis strategy.

[0358] Optionally, if the percentage of video frames in the target video clip that can be processed within the analysis time allocated to the target video clip is greater than a second preset threshold and less than a first preset threshold, then the target analysis strategy allocated to the target video clip is an intensive key segment analysis strategy. The second preset threshold is less than the first preset threshold. For example, the first preset threshold is 100%, and the second preset threshold is 60%.

[0359] For example, if the analysis time allocated to the target video material is 1.6 seconds, and the analysis time required to analyze all the video frames of the target video material is 2 seconds, then the percentage of video frames in the target video material that can be processed within the analysis time allocated to the target video material is 1.6 / 2 = 80%, which is less than the first preset threshold of 100% and greater than the second preset threshold of 60%. Therefore, the analysis strategy allocated to the target video material is the intensive key segment analysis strategy.

[0360] Optionally, if the percentage of video frames in the target video clip that can be processed within the analysis time allocated to the target video clip is greater than a third preset threshold and less than a second preset threshold, then the target analysis strategy allocated to the target video clip is a sparse key segment analysis strategy. The third preset threshold is less than the second preset threshold. For example, the third preset threshold can be a percentage obtained based on the minimum highlight duration, where the minimum highlight duration can refer to the shortest duration required to generate a highlight segment. The third preset threshold can be a percentage obtained by dividing the minimum highlight duration by the video length of the target video clip. The second preset threshold can be 60%.

[0361] For example, the minimum highlight duration t0 can be obtained by the following formula (12):

[0362] t0=min((t3+t1*1.2) / v, (t3+t2*0.5) / v)(12)

[0363] Where t0 refers to the minimum highlight duration, t3 refers to the minimum number of video frames required for analysis of a highlight band, t1 refers to the minimum duration of a highlight segment, and t2 refers to the maximum duration of a highlight segment. v refers to the video analysis rate.

[0364] For example, generating a highlight clip typically requires at least 3 video frames, therefore at least 3 video frames need to be analyzed to obtain a highlight band. In one possible case, the computational performance of the electronic device requires 0.2s for the duration t3 to analyze 3 video frames. The electronic device usually sets the duration range of the highlight clip, including the minimum and maximum values ​​of the highlight clip duration. For example, the duration range of the highlight band set by the electronic device is 1s-2s. That is to say, the minimum duration t1 of the highlight clip is 1s, and the maximum duration t2 of the highlight clip is 2s. The video analysis rate is 5. Therefore, the minimum highlight duration t0 can be determined by the above formula (12) = min((0.2+1*1.2) / 5,(0.2+2*0.5) / 5) = min(0.44, 0.6) = 0.44s. The third preset threshold can be 0.44 / 2 = 22%.

[0365] For example, if the analysis time allocated to the target video material is 1 second, and the analysis time required to analyze all the video frames of the target video material is 2 seconds, then the percentage of video frames in the target video material that can be processed within the analysis time allocated to the target video material is 1 / 2 = 50%, which is less than the second preset threshold of 60% and greater than the third preset threshold of 22%. Therefore, the analysis strategy allocated to the target video material is a sparse key segment analysis strategy.

[0366] Optionally, if the percentage of video frames in the target video clip that can be processed within the analysis time allocated to the target video clip is less than a third preset threshold, then the target analysis strategy allocated to the target video clip is an I-frame sampling analysis strategy. The third preset threshold can be a percentage obtained based on the minimum highlight duration.

[0367] For example, if the analysis time allocated to the target video material is 0.4s, and the analysis time required to analyze all the video frames of the target video material is 2s, then the percentage of video frames in the target video material that can be processed within the analysis time allocated to the target video material is 0.4 / 2 = 20%, which is less than the third preset threshold of 22%. Therefore, the target analysis strategy allocated to the target video material is the I-frame sampling analysis strategy.

[0368] In one possible scenario, the analysis strategy for each target video clip can be determined directly based on the analysis duration allocated to it and the analysis time required to analyze all the video frames. The analysis time required to analyze all the video frames can be obtained from the video duration and video analysis rate of the target video clip.

[0369] For example, after the solution interface layer obtains the target video material's video duration as 10s, the allocated analysis time as 1s, and the video analysis rate as 5, dividing the target video material's video duration of 10s by the video analysis rate of 5 yields the analysis time required to analyze all video frames of the target video material as 10 / 5 = 2s. Then, the analysis time allocated to the target video material and the analysis time required to analyze all video frames of the target video material are compared to determine the analysis strategy for the target video material.

[0370] Optionally, if the analysis duration allocated to the target video clip is greater than the analysis time required to analyze all video frames of the target video clip, the analysis strategy allocated to the target video clip is a full analysis strategy. For example, if the analysis duration allocated to the target video clip is 2.2 seconds, and the analysis time required to analyze all video frames in the target video clip is 2 seconds, then the target analysis strategy allocated to the target video clip is a full analysis strategy.

[0371] Optionally, if the analysis duration allocated to the target video material is less than the analysis time required to analyze all video frames of the target video material, but greater than the product of the analysis time required to analyze all video frames of the target video material and the fourth preset threshold, then the analysis strategy allocated to the target video material is a dense key segment analysis strategy. For example, if the analysis duration allocated to the target video material is 1.6s, the fourth preset threshold is 60%, and the analysis time required to analyze all video frames of the target video material is 2s, then the analysis duration of 1.6s allocated to the target video material is less than the analysis time required to analyze all video frames of the target video material (2s), but greater than the product of the analysis time required to analyze all video frames of the target video material (2s) and the fourth preset threshold (60%) (2*60%=1.2s), then the analysis strategy allocated to the target video material is a dense key segment analysis strategy.

[0372] Optionally, if the analysis time allocated to the target video material is less than the product of the analysis time required to analyze all video frames of the target video material and the fourth preset threshold, but greater than the fifth preset threshold, the analysis strategy allocated to the target video material is a sparse key segment analysis strategy. The fifth preset threshold can refer to the minimum highlight duration.

[0373] For example, the minimum highlight duration t0 can be obtained based on the following formula (13):

[0374] t0=min((t3+t1*1.2) / v, (t3+t2*0.5) / v)(13)

[0375] Where t0 refers to the minimum highlight duration, t3 refers to the minimum number of video frames required for analysis of a highlight band, t1 refers to the minimum duration of a highlight segment, and t2 refers to the maximum duration of a highlight segment. v can refer to the video analysis rate.

[0376] For example, generating a highlight clip typically requires at least 3 video frames, therefore at least 3 video frames need to be analyzed to obtain a highlight band. In one possible scenario, the computational performance of the electronic device requires 0.2 seconds to analyze 3 video frames, t3. Electronic devices usually set a range for the duration of highlight clips, including a minimum and a maximum duration. For example, the duration range of the highlight band set by the electronic device might be 1-2 seconds. That is, the minimum highlight clip duration t1 is 1 second, and the maximum highlight clip duration t2 is 2 seconds. The video analysis rate is 5. Therefore, based on the above formula, the minimum highlight duration t0 = min((0.2 + 1 * 1.2) / 5, (0.2 + 2 * 0.5) / 5) = min(0.44, 0.6) = 0.44 seconds. This means the fifth preset threshold is 0.44 seconds.

[0377] For example, if the analysis duration allocated to the target video clip is 1 second, the analysis time required to analyze all video frames of the target video clip is 2 seconds, the fourth preset threshold is 60%, and the fifth preset threshold is 0.44 seconds, then the analysis duration allocated to the target video clip is 1 second, which is less than the product of the analysis time required to analyze all video frames of the target video clip (2 seconds) and the fourth preset threshold (60%) (2 * 60% = 1.2 seconds), and greater than the fifth preset threshold (0.44). Therefore, the analysis strategy allocated to the target video clip is the I-frame sampling analysis strategy.

[0378] Optionally, if the analysis duration allocated to the target video clip is less than a fifth preset threshold, the target analysis strategy allocated to the target video clip is a sparse key segment analysis strategy. The fifth preset threshold can refer to the minimum highlight duration. For example, if the analysis duration allocated to the target video clip is 0.4s and the fifth preset threshold is 0.44s, the analysis duration of 0.4s is less than the fifth preset threshold of 0.44s; therefore, the analysis strategy allocated to the target video clip is an I-frame sampling analysis strategy.

[0379] Step 6069: The solution interface layer sends a video analysis command r12 to the pipeline interface layer. The video analysis command r12 carries the fd of a target video material and its corresponding analysis strategy indication information K.

[0380] The analysis strategy indication information K is used to uniquely indicate an analysis strategy.

[0381] In one example, when analyzing multiple target video clips, they can be analyzed one by one. That is, the solution interface layer first requests the analysis of one target video clip from the multiple target video clips (which can be any one of the multiple target video clips). Therefore, the video analysis instruction carries the descriptor identifier fd of a target video clip and the analysis strategy instruction information corresponding to this target video clip.

[0382] Optionally, the video analysis instruction r12 also includes frame indication information, which is used to indicate the position information of the corresponding video frame, such as information indicating which frame it is.

[0383] Step 6070: The pipeline interface layer decodes video frames based on the file descriptor (fd) of the target video material.

[0384] In implementation, the pipeline interface layer retrieves the target video material from the corresponding storage address based on the file descriptor (fd) of the target video material, and then performs video frame decoding on the target video material.

[0385] Step 6071: The pipeline interface layer stores the decoded data of the video frames into the shared buffer.

[0386] Step 6072: The pipeline interface layer sends a video frame compression instruction r13 to the resolution reduction interface of the service interface module 1. The video frame compression instruction r13 carries the frame data address d3 and identifier a of a video frame of the target video material.

[0387] The frame data address d3 is used to indicate the storage location of this video frame in the shared buffer.

[0388] Similar to the processing flow of the target image material, before analyzing the target video material, the resolution of each video frame in the target video material can be reduced sequentially through the underlying layer. Since the service interface module 1 in the service interface layer corresponds to the resolution reduction algorithm, the pipeline interface layer calls the resolution reduction interface of the service interface module 1 and sends the video frame compression command r13.

[0389] Optionally, the resolution of this video frame can also be carried in the video frame compression instruction r13.

[0390] Step 6073: In response to r13, service interface module 1 sends a video frame compression instruction r14 to the service transport layer. r14 includes identifiers a, d3, module identifier 1, and interface indication information tp3.

[0391] If the video frame compression instruction r13 also carries the resolution, the service interface module 1 can determine whether the video frame needs to be down-processed based on the resolution. For example, if the resolution is less than the resolution threshold, it is determined that no down-processing is needed; if the resolution is greater than or equal to the resolution threshold, it is determined that down-processing is needed. If down-processing is determined to be needed, the service interface module 1 sends the video frame compression instruction r14 to the service transport layer. If down-processing is determined not to be needed, the service interface module 1 can send a notification to the pipeline interface layer indicating that no down-processing is required.

[0392] Step 6074: The service transport layer sends a video frame compression command r14 to the service management layer.

[0393] Step 6075: The service management layer determines the service execution instance T1 based on identifier a and module identifier 1.

[0394] Step 6076: The service management layer sends a video frame compression command r14 to the service execution instance T1 of the service function layer.

[0395] The service management layer calls the service execution instance T1 of the service function layer and sends the video frame compression command r14 to the service execution instance T1.

[0396] Step 6077: The service function layer sends the video frame compression instruction r14 to the down-resolution algorithm module through the service execution instance T1 according to tp3.

[0397] Service execution instance T1 determines that resolution downscaling is required based on interface indication information tp3. Therefore, it calls the resolution downscaling algorithm's interface and sends the video frame compression command r14 to the interface. Afterward, service execution instance T1 listens for feedback from the resolution downscaling algorithm module through a callback listener.

[0398] Step 6078: In response to the video frame compression instruction r14, the resolution reduction algorithm module performs resolution reduction processing on the video frame based on d3.

[0399] As an example of this application, the down-resolution algorithm module reads a video frame from the shared buffer according to the frame data address d3 in the video frame compression instruction r14. Typically, the read video frame is in NV12 format. In this case, the video frame is decoded into bitmap format, and then the bitmap format video frame is down-resolution processed, for example, down-diminished to 480p frame data. Afterwards, the 480p frame data is converted back to NV12 format to obtain the low-resolution video frame.

[0400] Step 6079: The down-resolution algorithm module stores the obtained low-resolution video frames into the shared buffer.

[0401] Step 6080: The down-resolution algorithm module sends frame data address d4 to the service execution instance T1 of the service function layer. d4 is used to indicate the storage location of the low-resolution video frame in the shared buffer.

[0402] The resolution reduction algorithm module sends frame data address d4 to the service execution instance T1 of the service function layer according to the identifier a.

[0403] Step 6081: The service function layer returns frame data address d4 through service execution instance T1.

[0404] Service execution instance T1 listens for feedback from the down-resolution algorithm module via a callback listener. When it receives frame data address d4, it sends it to the service management layer. The service management layer then sends frame data address d4 to the service transport layer, which in turn sends it to service interface module 1. Service interface module 1 receives frame data address d4 through the down-resolution interface and sends it to the pipeline interface layer.

[0405] In one example, after the pipeline interface layer receives frame data address d4, it can delete the unprocessed frame video corresponding to frame data address d4 in the shared buffer.

[0406] Step 6082: The pipeline interface layer sends a video analysis command r15 to the video analysis interface of the service interface module 2. The video analysis command r15 carries the frame data address d4, the identifier a, the analysis strategy indication information K, and the frame indication information.

[0407] Frame indication information is used to determine whether a video frame is the last video frame of the target video material. For example, if the frame indication information is EOF (end of frame), it means that this video frame is the last video frame in the target video material. Otherwise, if the frame indication information is not EOF, it means that this video frame is not the last video frame in the target video material.

[0408] Step 6083: Service interface module 2 sends video frame analysis instruction r16 to the service transport layer. r16 carries d4, identifier a, analysis strategy indication information K, frame indication information, module identifier 2, and interface indication information tp4 of the video analysis interface.

[0409] Step 6084: The service transport layer sends a video frame analysis command r16 to the service management layer.

[0410] Step 6085: The service management layer determines the service execution instance T2 based on identifier a and module identifier 2.

[0411] For details on its implementation, please refer to the previous text.

[0412] Step 6086: The service management layer sends a video frame analysis command r16 to the service execution instance T2 of the service function layer.

[0413] Step 6087: The service function layer sends the video frame analysis instruction r16 to the highlight segment algorithm module through the service execution instance T2 according to tp4.

[0414] Service execution instance T2, based on tp4, calls the video analysis interface of the highlight fragment algorithm module and sends video frame analysis instruction 16 to the video analysis interface so that the highlight fragment algorithm module can perform video frame analysis.

[0415] Step 6088: In response to the video frame analysis instruction r16, the highlight segment algorithm module performs video frame analysis based on d4 and the analysis strategy indicated by the analysis strategy instruction information K.

[0416] In one example, the highlight fragment algorithm module retrieves low-resolution video frames from the shared buffer based on d4, and analyzes the low-resolution video frames using analysis measurements identified by the analysis strategy indication information K to determine whether they belong to highlight fragments.

[0417] Step 6089: If the video frame is determined to be not the last video frame of the target video material based on the frame indication information, the highlight fragment algorithm module sends an analysis success notification to the service execution instance T2 of the service function layer.

[0418] In other words, if the currently analyzed video frame is not the last video frame of the final target video material, the highlight fragment algorithm module will not return the analysis result after the video frame analysis is completed, but will instead record the analysis result of the processed video frame. To facilitate the pipeline interface layer to know that the analysis of this video frame has been completed, the highlight fragment algorithm module will return an analysis success notification to indicate that the analysis of this video frame has been completed.

[0419] Step 6090: The service function layer returns an analysis success notification through service execution instance T2.

[0420] In implementation, the service capability layer receives the analysis success notification through the callback listener of service execution instance T2 and sends it to the service management layer. The service management layer sends the analysis success notification to service interface module 2 through the service transport layer. Service interface module 2 receives the analysis success notification through the video analysis interface and sends it to the pipeline interface layer.

[0421] Step 6091: The pipeline interface layer continues to analyze the next video frame.

[0422] Step 6092: If the video frame is determined to be the last video frame of the target video material based on the frame indication information, the highlight fragment algorithm module sends the highlight fragment location information to the service execution instance T2 of the service function layer.

[0423] If the frame indication information determines that the currently analyzed video frame is the last video frame of the target video material, then after the analysis of this video frame is completed, the highlight fragment algorithm module sends the highlight fragment location information of this target video material to the service execution instance T2 of the service function layer according to the identifier 'a'. For example, the highlight fragment location information includes the start time point and the end time point.

[0424] Step 6093: The service function layer returns the highlight fragment location information through the service execution instance T2.

[0425] In implementation, the service capability layer receives the highlight fragment location information through the callback listener of service execution instance T2 and sends it to the service management layer. The service management layer sends the highlight fragment location information to service interface module 2 via the service transport layer using a callback. At this time, the parameter management interface of service interface module 2 is called back, thereby receiving the highlight fragment location information through the parameter management interface, and then sending the highlight fragment location information to the pipeline interface layer.

[0426] Step 6094: The pipeline interface layer sends the highlight fragment position information to the solution interface layer.

[0427] It should be noted that the embodiments in this application only take the example of requesting the underlying layer to process the target video material frame by frame. In another example, the pipeline interface layer can also request the underlying layer to process all video frames in the target video material at once. In this case, after decoding the video frames and storing them in the shared buffer, the pipeline interface layer can transmit the file descriptor (fd) of each video frame in the target video material to the underlying layer for down-resolution processing. Then, the underlying layer performs highlight segment analysis on each down-resolution video frame to determine the highlight segment location information. For the specific process, please refer to the processing flow of a single video frame.

[0428] Step 6095: If there are unanalyzed target video clips, the solution interface layer continues to process the next target video clip.

[0429] In other words, after the analysis of the target video material is completed, if there are other unanalyzed target video materials, the solution interface layer will continue to process the next target video material according to the above-mentioned target video material down-resolution process and highlight segment analysis process.

[0430] Step 6096: After all target video materials have been analyzed, the solution interface layer sends the highlight segment analysis results of all target materials to the application function layer.

[0431] The highlight fragment analysis results include the score value of each target image in multiple target image materials, and the highlight fragment location information of each video material in multiple target video materials.

[0432] Step 6097: The application function layer edits and filters the target material selected by the user based on the highlight fragment analysis results.

[0433] Step 6098: The application function layer sends the edited and filtered content to the basic capability layer.

[0434] Step 6099: The basic capability layer splices the content to obtain video clips.

[0435] For example, assuming that the content determined after editing and filtering includes target image material 1 (highlight clip 1), [20s, 30s] of target video material 1 (highlight clip 2), [5s, 10s] of target video material 2 (highlight clip 3) and [20s, 25s] (highlight clip 4), and [1s, 5s] of target video material 3 (highlight clip 5) and the 12th second (highlight clip 6), target image material 1, [20s, 30s] of target video material 1, [5s, 10s] and [20s, 25s] of target video material 2, and [1s, 5s] and the 12th second of target video material 3 can be spliced ​​together, that is, splicing highlight clip 1, highlight clip 2, highlight clip 3, highlight clip 4, highlight clip 5 and highlight clip 6.

[0436] Step 6100: The basic capability layer sends video clips to the video editing business layer.

[0437] Step 6101: The video editing business layer displays the video clip.

[0438] As an example of this application, after the application function layer edits and filters the target material based on the highlight fragment analysis results, it can also request the underlying layer to obtain the theme of the highlight fragment. Specifically, the application function layer calls the theme summary interface of the solution interface layer, sending the highlight fragment location information of the target video material to the theme summary interface. The solution interface layer sends the highlight fragment location information to the pipeline interface layer and instructs the pipeline interface layer to obtain the theme of the highlight fragment. At this time, the pipeline interface layer calls the theme analysis interface of service interface module 2, sending the highlight fragment location information and identifier 'a' to the theme analysis interface. Service interface module 2 sends the highlight fragment location information and identifier 'a' to the service transport layer through the theme analysis interface. The service transport layer sends the highlight fragment location information, module identifier 2, interface indication information of the theme analysis interface, and identifier 'a' to the service management layer. Thus, the service management layer calls the service execution instance T2 of the service function layer based on module identifier 2 and identifier 'a', sending identifier 'a', interface indication information of the theme analysis interface, and highlight fragment location information to the service execution instance T2. Service execution instance T2 invokes the topic analysis interface of the highlight fragment algorithm module according to the interface instruction information of the topic analysis interface. It sends an identifier 'a' and highlight fragment location information to the topic analysis interface of the highlight fragment algorithm module, enabling the highlight fragment algorithm module to determine the corresponding topic based on the highlight fragment location information. Subsequently, the highlight fragment algorithm module returns the topic to the pipeline interface layer based on the identifier 'a'. The pipeline interface layer then returns the topic to the application function layer through the solution interface layer. In this way, the application function layer sends the edited and filtered content, along with the topic, to the basic capability layer. This allows the basic capability layer to stitch the edited video footage and filtered image footage together and apply the topic to the stitched video clip.

[0439] In one example, multiple applications may need to use the algorithm implementation layer's algorithm by calling the media platform framework layer. To facilitate distinguishing business requests from different applications, each application can issue its application identifier upon its initial call to the media platform framework layer. After generating identifier 'a', the pipeline interface layer within the media platform framework layer can store the mapping between identifier 'a' and the application identifier. Furthermore, in the above process, when the algorithm implementation layer sends the processing result back to the pipeline interface layer through the algorithm service framework layer, it also returns identifier 'a'. Since the pipeline interface layer stores the mapping between application identifiers and identifier 'a', it can determine which application to send the processing result to after receiving the processing result from the algorithm implementation layer, based on identifier 'a' and the mapping between the application identifier and identifier 'a'.

[0440] In one example, after generating a video clip, the application layer can instruct the algorithm service framework layer to release resources. In implementation, the video editing business layer can instruct the media platform framework layer to call the resource release interfaces in each service interface module of the algorithm framework layer. Each service interface module then sends a resource release instruction to the service management layer through the service transport layer. This instruction carries an identifier 'a', a module identifier, and interface indication information for the resource release interface. The service management layer can then release resources according to the instruction. For example, it can unbind the identifier 'a' from each service execution instance, release the service module instance, and cancel the callback used to provide feedback on the highlight segment analysis results of the video material by calling the service deregistration module, thereby saving system resources.

[0441] In this embodiment, a media platform framework layer and an algorithm service framework layer are provided between the application and the algorithm implementation layer. The algorithm service framework layer provides multiple services, each capable of interfacing with a specific chip-level capability within the algorithm implementation. Thus, when the media platform framework layer receives a first instruction from the application, it retrieves the material data of the material to be analyzed based on the address indication information carried in the first instruction. Subsequently, the media platform framework layer sends the material data and target identification information to the service algorithm framework layer. The service algorithm framework layer, based on the target identification information, invokes the highlight fragment algorithm through the first service corresponding to the highlight fragment algorithm, thereby performing highlight fragment analysis on the material to be analyzed based on the material data. In other words, any application can invoke the algorithm of the algorithm implementation layer simply by calling the interface provided by the media platform framework layer, thereby improving the algorithm's openness.

[0442] It should be noted that the above embodiment uses the HAL layer's resolution reduction algorithm to illustrate the target material's resolution reduction process. In another example, the media platform framework layer can also perform the resolution reduction process on the target material. In this case, please refer to... Figure 10 The method may include:

[0443] Steps 1001 to 1004 can be found in [reference needed]. Figure 6 Steps 6001 to 6004 in the illustrated embodiment.

[0444] Step 1005: The solution interface layer sends initialization parameters to the pipeline interface layer. The initialization parameters include the algorithm name B of the specular fragment algorithm and the interface indication information tp1 of the initialization interface.

[0445] In other words, when the target material does not need to be down-resolution processed using the HAL layer down-resolution algorithm, the down-resolution algorithm initialization is not required. However, since the highlight fragment algorithm will be used for highlight fragment analysis later, the highlight fragment algorithm still needs to be initialized.

[0446] Step 1006: The pipeline interface layer calls the initialization interface L2 of the service interface module 2 corresponding to the algorithm name B, and sends the initialization instruction r3 to L2, with r3 carrying the identifier a.

[0447] Step 1007: Service interface module 2 sends an initialization command r3 to the service transport layer through initialization interface L2. r3 includes module identifier 2, tp1 and identifier a.

[0448] Step 1008: The service transport module sends an initialization command r3 to the service management module.

[0449] Step 1009: The service management layer creates a service execution instance T2 corresponding to service function module 2 based on module identifier 2.

[0450] Step 1010: The service management layer sends an initialization command r3 to the service execution instance T2 of the service function layer, and manages the identifier a and the service execution instance T2.

[0451] Step 1011: Initialize the service execution instance T2 in the service function layer.

[0452] Step 1012: The service function layer sends the initialization command r3 to the highlight fragment algorithm module through the service execution instance T2 according to tp1.

[0453] Step 1013: In response to the initialization command r3, the highlight fragment algorithm module performs algorithm initialization.

[0454] Step 1014: The highlight fragment algorithm module returns an initialization success notification b.

[0455] Step 1015: The pipeline interface layer calls the performance analysis interface L3 of service interface module 2 and sends a speed query command r4 to the performance analysis interface L3, with the identifier a carried in r4.

[0456] Step 1016: Service interface module 2 sends a speed query command r5 to the service transport layer through the performance analysis interface L3. r5 includes identifier a, module identifier 2 and interface indication information tp2.

[0457] Step 1017: The service transport layer sends a speed query command r5 to the service management layer.

[0458] Step 1018: Send r5 to service execution instance T2 based on identifier a and module identifier 2.

[0459] Step 1019: The service function layer calls the chip analysis speed interface of the specular fragment algorithm module through service execution instance 2 based on tp2.

[0460] Step 1020: The highlight fragment algorithm module sends the chip analysis speed to the service function layer.

[0461] Step 1021: The service function layer returns the chip analysis speed.

[0462] Step 1022: The pipeline interface layer sends the chip analysis speed to the solution interface layer.

[0463] Step 1023: Analysis speed of the solution interface layer memory chip.

[0464] Step 1024: The solution interface layer sends an initialization completion notification to the video editing business layer.

[0465] Step 1025: The video editing business layer sends a video processing speed query command r6 to the solution interface layer. The video processing speed query command r6 carries the file descriptor fd1 of the target video material to be queried.

[0466] Step 1026: The solution interface layer obtains the resolution and frame rate of the target video material based on fd1.

[0467] Step 1027: The solution interface layer determines the analysis speed of the target video material based on the chip analysis speed, resolution, and frame rate.

[0468] Step 1028: The solution interface layer feeds back the analysis speed to the video editing business layer.

[0469] Step 1029: If there are unqueried target video materials, the video editing business layer continues to query the analysis speed of the next target video material.

[0470] Step 1030: If there are no unqueried target video materials, the video editing business layer determines the total target analysis time based on the analysis speed of all target video materials and the preset analysis time of all target image materials.

[0471] Step 1031: The video editing business layer sends the material analysis command r7 to the solution interface layer. The material analysis command r7 carries the total target analysis duration and the file descriptor (fd) of each target material.

[0472] Next, the solution interface layer processes the footage using the specular fragment algorithm implemented in the algorithm layer. Please refer to [link / reference]. Figure 11 First, the target image material is analyzed:

[0473] Step 1032: In response to r7, the solution interface layer sends an image analysis command r8 to the pipeline interface layer, r8 carrying the fd2 of the target image material to be analyzed.

[0474] Step 1033: The pipeline interface layer obtains the target image material based on the fd2 of the target image material.

[0475] Step 1034: The pipeline interface layer performs resolution down-processing on the target image material to obtain low-resolution target image material.

[0476] For details on how the pipeline interface layer performs resolution downscaling on the target image material, please refer to the implementation of the resolution downscaling algorithm module.

[0477] Step 1035: The pipeline interface layer stores the low-resolution target image material into the shared cache.

[0478] Step 1036: The pipeline interface layer sends an image analysis instruction r10 to the image analysis interface of the service interface module 2. The image analysis instruction r10 carries the frame data address d2 and the identifier a. d2 is used to indicate the storage location of the low-resolution target image material in the shared cache.

[0479] Step 1037: In response to r10, service interface module 2 sends an image analysis instruction r11 to the service transport layer. r11 carries d2, identifier a, module identifier 2 and interface indication information tp4 of the image analysis interface.

[0480] Step 1038: The service transport layer sends an image analysis command r11 to the service management layer.

[0481] Step 1039: The service management layer determines the service execution instance T2 based on identifier a and service identifier 2.

[0482] Step 1040: The service management layer sends an image analysis command r11 to the service execution instance T2 of the service function layer.

[0483] Step 1041: The service function layer sends the image analysis instruction r11 to the highlight fragment algorithm module through the service execution instance T2 according to tp4.

[0484] Step 1042: In response to r11, the highlight fragment algorithm module determines the score value of the target image material based on the frame data address d2.

[0485] Step 1043: The highlight fragment algorithm module sends the score value to the service execution instance T2 of the service function layer.

[0486] Step 1044: The service function layer returns the score value through the service execution instance T2.

[0487] Step 1045: The pipeline interface layer sends the score value to the solution interface layer.

[0488] Step 1046: If there are unprocessed target image materials, the solution interface layer continues to process the target image materials of the next frame.

[0489] After analyzing all target image materials, the solution interface layer then uses the highlight fragment algorithm from the algorithm implementation layer to analyze each target video material. Finally, based on the highlight fragment analysis results of all target materials, a video clip is synthesized. For details, please refer to... Figure 12 It can include the following:

[0490] Step 1047: If there are no unprocessed target image materials, the solution interface layer determines the difference between the total target analysis time and the total image analysis time to obtain the expected time.

[0491] Step 1048: The solution interface layer queries the video duration of each target video material based on the file descriptor (fd) of each target video material.

[0492] Step 1049: The solution interface layer determines the analysis duration of each target video clip based on the expected duration and the video duration of each target video clip.

[0493] Step 1050: The solution interface layer determines the analysis strategy corresponding to each target video material based on the analysis duration of each target video material.

[0494] Step 1051: The solution interface layer sends a video analysis command r12 to the pipeline interface layer. The video analysis command r12 carries the fd of a target video material and its corresponding analysis strategy indication information K.

[0495] Step 1052: The pipeline interface layer decodes video frames based on the file descriptor (fd) of the target video material.

[0496] Step 1053: The pipeline interface layer performs down-resolution processing on the decoded data to obtain low-resolution video frames.

[0497] Step 1054: The pipeline interface layer stores low-resolution video frames into a shared buffer.

[0498] Step 1055: The pipeline interface layer sends a video analysis command r15 to the video analysis interface of the service interface module 2. The video analysis command r15 carries the frame data address d4, the identifier a, the analysis strategy indication information K, and the frame indication information. d4 is used to indicate the storage location of the low-resolution video frame in the shared buffer.

[0499] Step 1056: Service interface module 2 sends video frame analysis instruction r16 to the service transport layer. r16 carries d4, identifier a, analysis strategy indication information K, frame indication information, module identifier 2, and interface indication information tp4 of the video analysis interface.

[0500] Step 1057: The service transport layer sends a video frame analysis command r16 to the service management layer.

[0501] Step 1058: The service management layer determines the service execution instance T2 based on identifier a and module identifier 2.

[0502] Step 1059: The service management layer sends a video frame analysis command r16 to the service execution instance T2 of the service function layer.

[0503] Step 1060: The service function layer sends the video frame analysis instruction r16 to the highlight segment algorithm module through the service execution instance T2 according to tp4.

[0504] Step 1061: In response to the video frame analysis instruction r16, the highlight segment algorithm module performs video frame analysis based on d4 and the analysis strategy indicated by the analysis strategy instruction information K.

[0505] Step 1062: If the video frame is determined to be not the last video frame of the target video material based on the frame indication information, the highlight fragment algorithm module sends an analysis success notification to the service execution instance T2 of the service function layer.

[0506] Step 1063: The service function layer returns an analysis success notification through service execution instance T2.

[0507] Step 1064: The pipeline interface layer continues to analyze the next video frame.

[0508] Step 1065: If the video frame is determined to be the last video frame of the target video material based on the frame indication information, the highlight fragment algorithm module sends the highlight fragment location information to the service execution instance T2 of the service function layer.

[0509] Step 1066: The service function layer returns the highlight fragment location information through the service execution instance T2.

[0510] Step 1067: The pipeline interface layer sends the highlight fragment position information to the solution interface layer.

[0511] Step 1068: If there are unanalyzed target video clips, the solution interface layer continues to process the next target video clip.

[0512] Step 1069: After all target video footage has been analyzed, the solution interface layer sends the highlight segment analysis results of all target footage to the application function layer.

[0513] Step 1070: The application function layer edits and filters the target material selected by the user based on the highlight fragment analysis results.

[0514] All highlight fragment analysis results include the score value of each target image in multiple target image materials, and the highlight fragment location information of each video material in multiple target video materials.

[0515] Step 1071: The application function layer sends the edited and filtered content to the basic capability layer.

[0516] Step 1072: The basic capability layer splices the content to obtain video segments.

[0517] Step 1073: The basic capability layer sends video clips to the video editing business layer.

[0518] Step 1074: The video editing business layer displays the video clip.

[0519] In this embodiment, a media platform framework layer and an algorithm service framework layer are provided between the application and the algorithm implementation layer. The algorithm service framework layer provides multiple services, each capable of interfacing with a specific chip-level capability within the algorithm implementation. Thus, when the media platform framework layer receives a first instruction from the application, it retrieves the material data of the material to be analyzed based on the address indication information carried in the first instruction. Subsequently, the media platform framework layer sends the material data and target identification information to the service algorithm framework layer. The service algorithm framework layer, based on the target identification information, invokes the highlight fragment algorithm through the first service corresponding to the highlight fragment algorithm, thereby performing highlight fragment analysis on the material to be analyzed based on the material data. In other words, any application can invoke the algorithm of the algorithm implementation layer simply by calling the interface provided by the media platform framework layer, thereby improving the algorithm's openness.

[0520] Figure 13This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. This computer device can be the electronic device described in the various embodiments above. See also... Figure 13 The computer device includes at least one processor 1301, a communication bus 1302, a memory 1303, and at least one communication interface 1304.

[0521] The processor 1301 may be a microprocessor (including a central processing unit (CPU) or the like), an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0522] The communication bus 1302 may include a path for transmitting information between the aforementioned components.

[0523] The memory 1303 may be a read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), optical disc (including compact disc read-only memory (CD-ROM), compressed optical disc, laser disc, digital versatile optical disc, Blu-ray disc, etc.), magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 1303 may exist independently and be connected to the processor 1301 via the communication bus 1302. The memory 1303 may also be integrated with the processor 1301.

[0524] Communication interface 1304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0525] In a specific implementation, as one example, the processor 1301 may include one or more CPUs, such as Figure 13 CPU0 and CPU1 are shown in the diagram.

[0526] In a specific implementation, as one example, a computer device may include multiple processors, such as... Figure 13The processors 1301 and 1305 are shown in the diagram. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0527] In a specific implementation, as one embodiment, the computer device may further include an output device 1306, an input device 1307, and a camera 1308. The output device 1306 communicates with the processor 1301 and can display information in various ways. For example, the output device 1306 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 1307 communicates with the processor 1301 and can receive user input in various ways. For example, the input device 1307 may be a mouse, keyboard, touchscreen device, or sensing device, etc. The camera 1308 is used to capture images or video.

[0528] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of this application do not limit the type of computer device.

[0529] The memory 1303 stores program code 1310 for executing the scheme of this application, and the processor 1301 executes the program code 1310 stored in the memory 1303. This computer device can implement the calling algorithm methods provided in the above embodiments through the processor 1301 and the program code 1310 in the memory 1303.

[0530] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0531] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0532] The above-described embodiments are optional embodiments provided by this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for invoking an algorithm, characterized in that, The method is applied to an electronic device, which includes a first application, a media middleware framework layer, an algorithm service framework layer, and an algorithm implementation layer. The algorithm service framework layer provides multiple services, each of which interfaces with a chip-level capability in the algorithm implementation layer. The algorithm service framework layer includes a service interface layer, a service transmission layer, a service management layer, and a service function layer. The service interface layer includes at least one service interface module, and the service function layer includes at least one service function module. Each service interface module provides one service. The method includes: The media platform framework layer receives a first instruction sent by the first application, the first instruction including address indication information of the material to be analyzed; In response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed based on the address indication information; The media platform framework layer calls the first service interface module corresponding to the highlight fragment algorithm in the service interface layer, and sends a second instruction to the material analysis interface of the first service interface module. The first service interface module provides a first service. The second instruction includes the material data and target identification information. The target identification information is used to identify the instruction related to the current business request of the first application. In response to the second instruction, the first service interface module sends a third instruction to the service transport layer through the material analysis interface. The third instruction carries the material data, the target identification information, the first module identifier of the first service interface module, and the interface indication information of the material analysis interface. The service transport layer sends the third instruction to the service management layer; In response to the third instruction, the service management layer calls the highlight fragment algorithm through the first service function module in the service function layer that interfaces with the highlight fragment algorithm, based on the target identification information, the first module identification, and the interface instruction information of the material analysis interface, so as to perform highlight fragment analysis on the material to be analyzed based on the material data by means of the highlight fragment algorithm.

2. The method as described in claim 1, characterized in that, In response to the third instruction, the service management layer, based on the target identification information, the first module identification, and the interface instruction information of the material analysis interface, invokes the highlight fragment algorithm through the first service function module in the service function layer that interfaces with the highlight fragment algorithm, so as to perform highlight fragment analysis on the material to be analyzed based on the material data using the highlight fragment algorithm, including: In response to the third instruction, the service management layer determines the first service execution instance corresponding to the first service function module from the service function layer based on the target identification information, the first module identification, and the target binding relationship. The first service execution instance is a service execution instance that supports the first application in calling the highlight fragment algorithm. The target binding relationship includes identification information, module identification, and the binding relationship between service execution instances. The service management layer invokes the first service execution instance of the service function layer to send the interface instruction information of the material analysis interface and the material data to the first service execution instance; The service function layer transmits the material data to the highlight fragment algorithm through the first service execution instance according to the interface instruction information, so that the highlight fragment algorithm performs highlight fragment analysis on the material to be analyzed based on the material data.

3. The method as described in claim 1 or 2, characterized in that, Before the media middleware framework layer receives the first instruction sent by the first application, it also includes: The first application receives a user's selection of the material to be analyzed and confirmation to execute the one-click video creation function, which is used to generate a video clip including highlight segments based on the material to be analyzed. In response to the operation, the first application calls the initialization interface related to the one-click video creation function in the media middleware framework layer; The media middleware framework layer generates the target identification information and sends an algorithm initialization instruction to the first service in the algorithm service framework layer. The algorithm initialization instruction carries the target identification information. The algorithm service framework layer initializes the highlight fragment algorithm through the first service; The algorithm service framework layer binds the target identification information to the first service.

4. The method as described in claim 3, characterized in that, The media middleware framework layer includes a design interface layer and a channel interface layer, and the design interface layer includes the initialization interface. The media middleware framework layer sends an algorithm initialization instruction to the first service in the algorithm service framework layer, including: The design interface layer sends a fourth instruction to the channel interface layer, the fourth instruction including the algorithm name of the highlight fragment algorithm and the interface indication information of the initialization interface of the first service interface module; In response to the fourth instruction, the channel interface layer calls the initialization interface in the first service interface module and sends the algorithm initialization instruction to the initialization interface of the first service interface module according to the algorithm name and the interface indication information of the initialization interface of the first service interface module.

5. The method as described in claim 4, characterized in that, The algorithm service framework layer initializes the highlight fragment algorithm through the first service, including: The first service interface module sends a fifth instruction to the service transport layer. The fifth instruction includes the target identification information, the first module identifier, and the interface indication information of the initialization interface of the first service interface module. The service transport layer sends the fifth instruction to the service management layer; The service management layer creates a first service execution instance corresponding to the first service function module based on the first module identifier; The service management layer invokes the first service execution instance and sends the interface indication information of the initialization interface of the first service interface module to the first service execution instance; The service function layer initializes the first service execution instance, and the first service execution instance initializes the highlight fragment algorithm according to the interface instruction information of the initialization interface of the first service interface module.

6. The method as described in claim 5, characterized in that, The algorithm service framework layer binds the target identification information to the first service, including: The service management layer binds the target identification information, the first module identification, and the first service execution instance.

7. The method as described in claim 3, characterized in that, In response to the third instruction, the service management layer, based on the target identification information, the first module identification, and the interface indication information of the material analysis interface, calls the highlight fragment algorithm through the first service function module in the service function layer that interfaces with the highlight fragment algorithm. After the highlight fragment algorithm performs highlight fragment analysis on the material to be analyzed based on the material data, the process further includes: The algorithm service framework layer obtains the highlight fragment analysis results through the first service; The algorithm service framework layer sends the highlight fragment analysis results to the media middleware framework layer; The media middleware framework layer sends the highlight clip analysis results to the first application. The first application edits the material to be analyzed based on the highlight fragment analysis results to generate the video fragment.

8. The method as described in claim 7, characterized in that, The material to be analyzed includes target video material; The algorithm service framework layer obtains the highlight fragment analysis results through the first service, including: The service function layer receives the highlight fragment analysis results through the first service execution instance; The algorithm service framework layer sends the highlight segment analysis results to the media middleware framework layer, including: The service function layer sends the highlight fragment analysis results to the service management layer; The service management layer calls back to the first service interface module through the transport layer to send the highlight fragment analysis results to the first service interface module; The first service interface module sends the highlight fragment analysis results to the media platform framework layer.

9. The method as described in claim 1 or 2, characterized in that, In response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed based on the address indication information, including: In response to the first instruction, the media platform framework layer retrieves the material to be analyzed from the corresponding storage address according to the address indication information; The media platform framework layer stores the material to be analyzed in a shared cache, which is a cache shared by the media platform framework layer and the algorithm implementation layer. The media platform framework layer determines the frame data address of the material to be analyzed in the shared cache, and the material data is the frame data address of the material to be analyzed in the shared cache.

10. The method as described in claim 9, characterized in that, In response to the first instruction, after the media platform framework layer obtains the material to be analyzed from the corresponding storage address according to the address indication information, it further includes: The media platform framework layer performs resolution reduction processing on the material to be analyzed to obtain low-resolution material; The media middleware framework layer stores the media to be analyzed in a shared cache, including: The media middleware framework layer stores the low-resolution materials in the shared cache; The media platform framework layer determines the frame data address of the material to be analyzed in the shared cache, wherein the material data is the frame data address of the material to be analyzed in the shared cache, including: The media platform framework layer determines the frame data address of the low-resolution material in the shared cache, and the material data is the frame data address of the low-resolution material in the shared cache.

11. The method as described in claim 1 or 2, characterized in that, In response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed based on the address indication information, including: In response to the first instruction, the media platform framework layer retrieves the material to be analyzed from the corresponding storage address according to the address indication information; The media platform framework layer stores the material to be analyzed in a shared cache, which is a cache shared by the media platform framework layer and the algorithm implementation layer. The media middleware framework layer sends a material compression instruction to the algorithm service framework layer. The material compression instruction carries the frame data address of the material to be analyzed in the shared cache and the target identification information. In response to the material compression instruction, the algorithm service framework layer, based on the target identification information, calls the resolution reduction algorithm of the algorithm implementation layer through the second service, so as to perform resolution reduction processing on the material to be analyzed based on the frame data address of the material to be analyzed in the shared cache. The algorithm service framework layer obtains the frame data address of the low-resolution material in the shared cache through the second service. The low-resolution material is the data after the material to be analyzed has been down-resolution processed by the down-resolution algorithm. The algorithm service framework layer sends the frame data address of the low-resolution material in the shared cache to the media platform framework layer; The media middleware framework layer determines the frame data address of the low-resolution material in the shared cache as the material data.

12. The method as described in claim 11, characterized in that, The media middleware framework layer sends material compression instructions to the algorithm service framework layer, including: The media platform framework layer calls the second service interface module in the service interface layer that corresponds to the resolution reduction algorithm, and sends the material compression command to the resolution reduction interface of the second service interface module; In response to the material compression command, the algorithm service framework layer, based on the target identification information, calls the resolution reduction algorithm of the algorithm implementation layer through the second service to perform resolution reduction processing on the material to be analyzed based on the frame data address of the material to be analyzed in the shared cache, including: In response to the material compression command, the second service interface module sends a sixth command to the service transport layer through the resolution reduction interface. The sixth command carries the frame data address of the material to be analyzed in the shared cache, the target identification information, the second module identifier of the second service interface module, and the interface indication information of the resolution reduction interface. The service transport layer sends the sixth instruction to the service management layer; In response to the sixth instruction, the service management layer, based on the target identification information, the second module identification, and the interface indication information of the resolution reduction interface, calls the resolution reduction algorithm through the second service function module in the service function layer that interfaces with the resolution reduction algorithm, so as to perform resolution reduction processing on the material to be analyzed based on the frame data address of the material to be analyzed in the shared cache.

13. The method as described in claim 11, characterized in that, Before the media middleware framework layer sends the material compression instruction to the algorithm service framework layer, it also includes: The media middleware framework layer sends an algorithm initialization command to the second service in the algorithm service framework layer, and the algorithm initialization command carries the target identification information. The algorithm service framework layer initializes the resolution reduction algorithm through the second service; The algorithm service framework layer binds the target identification information to the second service.

14. The method as described in claim 1 or 2, characterized in that, The media platform framework layer includes a highlight segment analysis interface and a strategy monitoring interface; the media platform framework layer receives a first instruction sent by the first application, including: The first application sends the first instruction to the highlight fragment analysis interface; The highlight fragment analysis interface sends the first instruction to the strategy monitoring interface; In response to the first instruction, the media platform framework layer obtains the material data of the material to be analyzed based on the address indication information, including: In response to the first instruction, the strategy monitoring interface obtains the material data of the material to be analyzed based on the address indication information.

15. An electronic device, the structure of which includes a processor and a memory; The memory is used to store a program that supports the electronic device in executing the calling algorithm provided in any one of claims 1-14, and to store data involved in implementing the calling algorithm provided in any one of claims 1-14; The processor is configured to execute programs stored in the memory.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-14.

Images