Multimedia processing method and apparatus, and device and medium

Abstract

The embodiments of the present disclosure relate to a multimedia processing method and apparatus, and a device and a medium. The method comprises: in response to the start of a target operation on a target material in a multimedia editing window, displaying on a first canvas component an image of the target material, and displaying on a second canvas component a composite image of materials other than the target material, wherein the target material is a material to which an editing operation of image effect processing is applied, and the target operation is an operation of changing the display state of the target material; placing the first canvas component on the second canvas component, and adjusting the display state of the first canvas component on the second canvas component on the basis of an action of changing the display state of the target material during an operation process of the target operation; and in response to the end of the target operation, on the basis of the corresponding display state of the first canvas component at the end of the target operation and layer levels of all materials to be rendered, rendering all the materials to be rendered, so as to obtain a preview image.

Description

Multimedia processing method, device, equipment and medium

[0001] Cross-reference to Related Applications

[0002] The present application claims priority from the Chinese patent application No. 202410874714.9 and titled "Multimedia processing method, device, equipment and medium", filed on July 1, 2024, the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the field of multimedia technology, and in particular, to a multimedia processing method, device, equipment and medium. BACKGROUND

[0004] In a multimedia editing scenario, if a user performs a drag operation on a material, a frame refresh process is usually required, that is, all materials need to be rendered and refreshed during the operation. SUMMARY

[0005] To solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a multimedia processing method, device, equipment and medium.

[0006] The present disclosure provides a multimedia processing method, which comprises: in response to a start of a target operation on a target material in a multimedia editing window, displaying an image of the target material on a first canvas component and displaying a composite image of materials other than the target material on a second canvas component; wherein the target material is a material subjected to an editing operation of image effect processing, the target operation is an operation of changing the display state of the target material, and the materials other than the target material are materials other than the target material in all to-be-rendered materials corresponding to a target preview position; placing the first canvas component on the second canvas component and adjusting the display state of the first canvas component on the second canvas component based on an action of changing the display state of the target material during the operation process of the target operation; and in response to an end of the target operation, rendering the all to-be-rendered materials based on the display state of the first canvas component corresponding to the end of the target operation and the layer level of the all to-be-rendered materials to obtain a preview image.

[0007] Optionally, the method further comprises: deleting the first canvas component after the target material is rendered.

[0008] Optionally, the displaying of the composite image of the materials other than the target material on the second canvas component comprises: hiding the target material, rendering a composite image of the materials other than the target material, and displaying the composite image on the second canvas component.

[0009] Optionally, the hiding the target material comprises: changing a current coordinate of the target material to a coordinate outside the second canvas component; or setting the target material to be fully transparent.

[0010] Optionally, the adjusting the display state of the first canvas component on the second canvas component based on the action of changing the display state of the target material in the target operation comprises: obtaining an operation type of the target operation; wherein the operation type comprises a drag operation, a rotation operation or a zoom operation; determining a target key point of the first canvas component based on the operation type of the target operation; obtaining the action of changing the display state of the target material in the target operation according to a preset time interval during the operation of the target operation, and determining a target coordinate of the target key point of the first canvas component on the second canvas component based on the action of changing the display state of the target material in the target operation, and adjusting the position and / or size of the first canvas component on the second canvas component according to the target coordinate.

[0011] Optionally, the determining the target key point of the first canvas component based on the operation type of the target operation comprises: in a case where the operation type of the target operation is the drag operation, determining the target key point of the first canvas component to be a specified vertex of the first canvas component; in a case where the operation type of the target operation is the rotation operation or the zoom operation, determining the target key point of the first canvas component to be a center point of the first canvas component.

[0012] Optionally, a display state of the target material after rendering on the preview image is consistent with a corresponding display state of the first canvas component at the end of the target operation; wherein the display state is represented based on position and / or size.

[0013] The embodiment of the present disclosure further provides a multimedia processing device, comprising: a first rendering module, configured to render materials other than a target material on a current canvas presented by a multimedia editing window in response to a start of a target operation on the target material; wherein the target material is a material affected by a specific algorithm material corresponding to the multimedia editing window, the target operation is an operation of changing a display state of the target material; the materials other than the target material are materials other than the target material in all to-be-rendered materials corresponding to the multimedia editing window; a component creating module, configured to acquire an image of the target material, and create a first canvas component corresponding to the image of the target material to load the image of the target material to the first canvas component; a component adjusting module, configured to place the first canvas component loaded with the image of the target material on a second canvas component loaded with the current canvas, and adjust a display state of the first canvas component on the second canvas component based on an operation process of the target operation; and a second rendering module, configured to render all to-be-rendered materials corresponding to the multimedia editing window on the current canvas in response to an end of the target operation based on a display state of the first canvas component corresponding to the end of the target operation and layer levels of all to-be-rendered materials corresponding to the multimedia editing window.

[0014] The embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the multimedia processing method provided by the embodiment of the present disclosure.

[0015] The embodiment of the present disclosure further provides a computer readable storage medium, which stores a computer program, and the computer program is used to execute the multimedia processing method provided by the embodiment of the present disclosure.

[0016] The embodiment of the present disclosure further provides a computer program product, comprising a computer program, and the computer program is used to implement the multimedia processing method provided by the embodiment of the present disclosure when executed by a processor.

[0017] The above technical solutions and methods provided by the embodiment of the present disclosure can display the image of the target material on the first canvas component and display the materials other than the target material on the second canvas component at the start of the target operation on the target material (the material subjected to the image effect processing of the editing operation), and during the target operation, no material rendering is needed on the canvas, and only the display state of the first canvas component needs to be adjusted based on the operation process of the target operation, and at the end of the target operation, all to-be-rendered materials are rendered based on the display state of the first canvas component corresponding to the end of the target operation and the layer levels of all to-be-rendered materials.

[0018] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor are they used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure, together with the description.

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced below. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0021] FIG. 1 is a material dragging schematic diagram provided by an embodiment of the present disclosure;

[0022] FIG. 2 is an editing interface schematic diagram provided by an embodiment of the present disclosure;

[0023] FIG. 3 is an editing interface schematic diagram provided by an embodiment of the present disclosure;

[0024] FIG. 4 is a dragging rendering schematic diagram provided by the related art;

[0025] FIG. 5 is a dragging rendering schematic diagram provided by the related art;

[0026] FIG. 6 is a dragging rendering schematic diagram provided by the related art;

[0027] FIG. 7 is a track schematic diagram provided by an embodiment of the present disclosure;

[0028] FIG. 8 is a rendering effect schematic diagram provided by an embodiment of the present disclosure;

[0029] FIG. 9 is a rendering effect schematic diagram provided by an embodiment of the present disclosure;

[0030] FIG. 10 is a rendering effect schematic diagram provided by an embodiment of the present disclosure;

[0031] FIG. 11 is a flow schematic diagram of a multimedia processing method provided by an embodiment of the present disclosure;

[0032] FIG. 12 is a rendering effect schematic diagram provided by an embodiment of the present disclosure;

[0033] FIG. 13 is a canvas component schematic diagram provided by an embodiment of the present disclosure;

[0034] FIG. 14 is a schematic view of a canvas component according to an embodiment of the present disclosure;

[0035] FIG. 15 is a schematic view of a rendering effect according to an embodiment of the present disclosure;

[0036] FIG. 16 is a schematic view of a multimedia processing device according to an embodiment of the present disclosure;

[0037] FIG. 17 is a schematic view of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0039] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other different manners from those described herein; obviously, the embodiments described in the specification are only some embodiments of the present disclosure, not all embodiments.

[0040] In a multimedia editing scenario, a user can perform operations such as dragging, rotating, scaling, etc. on materials according to requirements to change the current display state of the materials. For ease of understanding, taking a dragging operation as an example, a schematic view of material dragging is shown in FIG. 1, which shows that a rendering canvas is presented in a clip window (which can also be referred to as a multimedia editing window), and the dragging operation can drag a target material from the lower left corner of the rendering canvas to the upper right corner (this dragging manner is only an example, and the dragging direction and the dragging length can be flexibly set according to requirements in actual application), at this time, the dragging original coordinates (x1, y1) of the top left corner of the target material are changed to the dragging new coordinates (x2, y2). During the dragging, the materials on the current entire canvas need to be refreshed constantly. If an algorithm material (such as a special effect) that needs to act on one or more other materials such as an image is involved, the single refresh rendering time is usually long due to the influence of the algorithm level, and the visual fluency during the execution of the dragging operation is poor, and the user experience is poor.

[0041] If there is special effect or other algorithmic material in the material that needs to act on other materials, it usually takes more rendering time. Especially for global algorithms (such as global special effects), it is more complex to render and takes more time to render when it acts on one or more other materials to make the other materials present the corresponding effect of the algorithm. In practical applications, the multimedia editing interface includes a multimedia editing window and a plurality of track axes corresponding to the multimedia editing window, each track axis corresponds to a material or a special effect or algorithm for image effect processing, the material types include but are not limited to images, videos, stickers, etc., each track axis corresponds to a layer, and the global algorithm refers to an algorithm that can act on all track axis materials located below the track axis it corresponds to. For ease of understanding, refer to the editing interface diagram shown in FIG. 2, which illustrates a clip window and track axes located below the clip window. In FIG. 2, three tracks corresponding to the clip window are simply illustrated before adding special effects / algorithm, namely track axis 1 to track axis 3, specifically, from bottom to top, they are material 1 track, material 2 track, and blank track axis. Accordingly, refer to the editing interface diagram shown in FIG. 3, which illustrates that the user has added a special effect / algorithm to track axis 3, and the special effect / algorithm corresponding to track axis 3 is a global algorithm. Since the track axis is located above the material 1 track axis and the material 2 track axis, the special effect / algorithm can be directly applied to material 1 and material 2. Each track corresponds to a corresponding material and a layer, for example, in FIG. 3, the layer level corresponding to material 1 is layer 1, the layer level corresponding to material 2 is layer 2, and the layer level corresponding to the special effect / algorithm is layer 3. As shown in FIG. 3, the special effect / algorithm is located at the top layer and has the highest layer level, and can directly act on material 1 and material 2 which have relatively low layer levels.

[0042] For ease of understanding, the embodiments of the present disclosure first provide a rendering example adopted by the related technology for the drag operation. Specifically, the drag rendering schematic diagrams shown in FIGS. 4-6 are referred to, which all take the example that the clip window only corresponds to material 1 and special effect / algorithm. FIGS. 4-6 simply show two track axes, which correspond to two layers in total, namely, material layer and special effect / algorithm layer, wherein the special effect / algorithm is the uppermost layer. FIG. 4 shows the rendering state at the beginning of the drag, the vertex coordinate of the upper left corner of material 1 is the start coordinate, FIG. 5 shows the rendering state during the drag, the vertex coordinate of the upper left corner of material 1 is the process coordinate, and FIG. 6 shows the rendering state at the end of the drag, the vertex coordinate of the upper left corner of material 1 is the end coordinate. From the beginning of the drag to the end of the drag, the related technology needs to constantly frame and refresh rendering, and needs to complete on-screen rendering for all materials each time. For ease of understanding, the case of three track axes is further provided, which can be referred to the track schematic diagram shown in FIG. 7. On the basis of FIGS. 4-6, track 3 is newly added, that is, material 2 not affected by the special effect / algorithm is newly added. FIG. 8 further shows a rendering effect schematic diagram, which shows the rendering effect corresponding to the track axes shown in FIG. 7, that is, the material arrangement logic of the track axes shown in FIG. 7 is presented, wherein the material 1 layer is located at the lowermost layer, the special effect / algorithm layer is located at the middle layer, and can affect material 1; the material 2 layer is located at the uppermost layer. It should be noted that during the drag of material 1 in the related technology, the current entire canvas is constantly refreshed and rendered, that is, all materials contained in the track at the current moment, such as images, stickers, special effects / algorithm, etc. need to be re-rendered all over again. The specific refresh and rendering logic is as follows: first, the materials corresponding to the track axes are rendered to the canvas in the order from bottom to top according to the track axes, for example, in FIGS. 7 and 8, the material 1 layer needs to be rendered first, and then the special effect / algorithm is rendered. The rendering logic of the special effect / algorithm includes that the effect of the special effect / algorithm is applied to all layers below the layer corresponding to the special effect / algorithm, that is, the special effect / algorithm needs to be applied to all materials corresponding to the track axes below the track axis corresponding to the special effect / algorithm. Taking the special effect / algorithm as an example, the material 1 will have a stroke effect.

[0043] However, when the user performs the operation of changing the material display state such as dragging the material, if the above-mentioned manner provided by the related technology is adopted, continuous refreshing of the rendering canvas will be triggered, and all the materials to be rendered on the track axis at the current time (i.e., the current frame) need to be re-rendered and uploaded to the screen once every time the refreshing is performed. Assuming that the current time involves 5 materials to be rendered, and the rendering processing of each material on the screen needs 5 ms, the total uploading (i.e., the end-to-end change of the canvas) rendering time is 25 ms. If the time of one-time full-screen rendering is 25 ms, the highest frame rate within 1000 ms is only 40 fps, and the user's perception is not good. For the convenience of understanding the full-screen rendering, taking the three track axes shown in FIG. 7 as an example, the rendering effect diagrams shown in FIGS. 9, 10 and 8 can be referred to. FIG. 9 illustrates that the material 1 corresponding to the track axis 1 needs to be rendered first, FIG. 10 illustrates that the special effect / algorithm corresponding to the track axis 2 is rendered on the basis of the rendered material 1, at this time, the special effect / algorithm has been applied to the material 1, so that the material 1 presents the effect corresponding to the special effect / algorithm, and FIG. 8 further illustrates that the material 2 corresponding to the track axis 3 is rendered on the basis of the layer corresponding to the special effect / algorithm. Since the layer of the material 2 is located above the layer of the special effect / algorithm, the material 2 will not be affected by the special effect / algorithm. Through the above-mentioned manner, full-screen rendering of all materials needs to be performed during the entire dragging process, the time consumption of one-time full-screen rendering is relatively long, and the frame rate (refresh rate) is relatively low.

[0044] To improve the above problems, the embodiments of the present disclosure provide a multimedia processing method, device, equipment and medium, which are described in detail as follows:

[0045] FIG. 11 is a flow diagram of a multimedia processing method provided by an embodiment of the present disclosure. The method can be executed by a multimedia processing device, which can be implemented by software and / or hardware and can be integrated into an electronic device. As shown in FIG. 11, the method mainly includes the following steps S1102-S1106:

[0046] In step S1102, in response to the start of the target operation on the target material in the multimedia editing window, the image of the target material is displayed on the first canvas component, and the synthesized image of the materials except the target material is displayed on the second canvas component.

[0047] The target material is a material to which an image effect processing is applied in an editing operation, specifically, the target material is a material to which an image effect processing is applied in an editing operation in an editing draft, the editing draft includes one or more materials and editing information for the materials. The target operation is an operation for changing the display state of the target material, the embodiments of the present disclosure do not limit the target operation, and any operation that can change the display state of the target material can be the target operation, such as a drag operation, a rotation operation, a zoom operation, etc., and the display state of the target material can be represented by the position and / or size of the target material, etc.

[0048] The material other than the target material is a material other than the target material among all the materials to be rendered corresponding to the target preview position. The material to be rendered mentioned in the embodiments of the present disclosure includes but is not limited to an image, a video, a text, etc., and mainly refers to all the materials to be rendered corresponding to the target preview position of the editing timeline, the target preview position can be specified by the user according to the needs, and the corresponding materials to be rendered can be the same or different at different preview positions of the editing timeline. In the embodiments of the present disclosure, the multimedia editing window corresponds to a plurality of track axes, different track axes correspond to different materials or special effects or algorithms used for image effect processing, etc., the multimedia editing window can be used to display the preview image of the material corresponding to different preview positions of the editing timeline, and the multimedia editing window can also be understood as the window of the preview player. The plurality of track axes are arranged in a bottom-up order, the material corresponding to the track axis located at the top has a higher layer level than the material corresponding to the track axis located at the bottom; and the materials corresponding to the track axes located below the track axis of the special effect or algorithm are all materials to which an image effect processing is applied in an editing operation, that is, the materials corresponding to the track axes located below the track axis of the special effect or algorithm all apply the special effect or algorithm, and the track axis corresponding to the target material is located below the track axis corresponding to the special effect or algorithm, in other words, the layer level of the target material is lower than the layer level of the special effect or algorithm, and the target material needs to be affected by the special effect or algorithm, that is, in the embodiments of the present disclosure, the target material specifically refers to a material affected by the special effect or algorithm and subjected to the target operation, and in actual application, the number of target materials can be one or more, which is not limited here.

[0049] In a specific implementation, in the embodiments of the present disclosure, a multimedia editing interface can be displayed for a user, which includes a multimedia editing window (such as the aforementioned clip window) and a track area; the track area can include one or more track axes, and specifically, a track axis can be established for each of the selected material and the special effect or algorithm set by the user for image effect processing, and the duration of the material in the multimedia editing window and the like can be adjusted through the track axis corresponding to each material, and the position of the track axis corresponds to the layer level of the material on the canvas, for example, the plurality of track axes are arranged from bottom to top, and the lower the position of the track axis, the lower the layer level of the material corresponding to the track axis. The multimedia editing window can present the rendering content of the target canvas, and the target canvas renders the material selected by the user at the target preview position. In the related art, all materials to be rendered are usually rendered, but in the present application, in order to optimize the rendering process during dragging, only the materials other than the target material are rendered at the beginning of dragging.

[0050] The embodiments of the present disclosure can create a corresponding first canvas component for the target material and a corresponding second canvas component for the entire draft canvas, that is, the rendering canvas corresponding to the editing draft. The size of the first canvas component is consistent with the size of the image of the target material, and the size of the second canvas component is consistent with the size of the draft canvas. The above-mentioned canvas component can be a canvas component of a webpage, specifically, a Canvas Dom. After creating the first canvas component, the image of the target material can be loaded through the first canvas component, and the picture content of the image of the target material can be displayed through the first canvas component. The embodiments of the present disclosure do not limit the image format of the target material, for example, the image of the target material can be an RGB image, a YUV image or an RGBA image of the target material. In some specific implementation examples, the image of the target material is an RGBA image of the target material, which can also be referred to as image RGBA data of the target material, and specifically includes four-channel data presentation, including R (red) channel data, G (green) channel data, B (blue) channel data and Alpha (transparency) channel data. In addition, the second canvas component set by the embodiments of the present disclosure only displays the synthesized image of the materials other than the target material, and before this, the materials other than the target material can be rendered first, and then displayed on the second canvas component. In actual application, the image corresponding to the material can be directly rendered and displayed on the canvas component, or the image corresponding to the material can be first rendered on the canvas, and then loaded and displayed on the canvas component corresponding to the canvas. The specific implementation can be flexibly set, which is not limited herein. It should be noted that the related art directly renders all materials on the target canvas, that is, the entire draft canvas, while the embodiments of the present disclosure set two canvas components to display the image of the target material and the synthesized image of the materials other than the target material, respectively, and the two canvas components are independent of each other, and the target material will not affect the display of other materials when the target operation is performed.

[0051] In step S1104, the first canvas component is placed on the second canvas component, and the display state of the first canvas component on the second canvas component is adjusted based on the action of changing the display state of the target material in the operation process of the target operation. The size of the first canvas component is consistent with the size of the image of the target material, and the content of the image of the target material can be directly displayed through the first canvas component. The size of the second canvas component is consistent with the size of the draft canvas, and the content of the synthesized image of the materials other than the target material can be directly displayed through the second canvas component. The display state of the first canvas component on the second canvas component includes the position and / or size of the first canvas component on the second canvas component. For example, the coordinate position of the first canvas component on the second canvas component is adjusted based on the operation process of the target operation. Since the first canvas component is placed on the second canvas component, the display content of the first canvas component is superimposed on the display content of the second canvas component, and the overall preview effect of all the materials to be rendered can be clearly presented to the user.

[0052] It should be noted that the first canvas component is independent of the second canvas component, and adjusting the display state of the first canvas component on the second canvas component does not affect the second canvas component, which is equivalent to not changing the state and display content of the second canvas component, but only changing the coordinate position of the first canvas component and other display states. In this way, when the target material is subjected to a drag operation or other operations that change the display state of the target material, it is not necessary to constantly refresh and render all the materials on the draft canvas, i.e., it is not necessary to constantly perform full-screen rendering operations on the draft canvas. Moreover, since the first canvas component is independent of the second canvas component during the target operation, the display content of the second canvas component is not changed, and therefore it is not necessary to render the specified algorithm material, i.e., the global algorithm is not applied to the target material during this period. Therefore, compared with related technologies, the rendering time during the target operation can be greatly reduced, and the performance during the target operation can be optimized.

[0053] In step S1106, in response to the end of the target operation, all the materials to be rendered are rendered based on the display state of the first canvas component corresponding to the end of the target operation and the layer levels of all the materials to be rendered corresponding to the multimedia editing window, to obtain a preview image. In actual applications, all the materials to be rendered can be directly rendered on the draft canvas, or the synthesized image of all the materials to be rendered can be rendered and displayed on the second canvas component, which is not limited herein. In addition, the first canvas component and the second canvas component can be deleted after rendering, or only the first canvas component can be deleted and the second canvas component can be continued to be retained, which can be flexibly set.

[0054] In actual application, the display state of the rendered target material on the preview image is consistent with the display state of the first canvas component corresponding to the end of the target operation; wherein the display state is characterized based on position and / or size. That is, the coordinate position of the first canvas component corresponding to the end of the target operation, that is, the coordinate position of the target material on the draft canvas at the end of the target operation, at which time the target material can be rendered based on the coordinate position. It should be noted that all materials to be rendered corresponding to the multimedia editing window need to be rendered in order based on the respective layer levels, so as to achieve the effect of full-screen rendering refreshing the canvas, and when rendering to the target material, the display state of the target material on the draft canvas at the end of the target operation (such as the position coordinate) can be obtained based on the display state of the first canvas component corresponding to the end of the target operation, so as to render the target material based on the display state.

[0055] In summary, compared with the related art, which needs to constantly render and refresh all materials during the target operation, the above method provided by the embodiments of the present disclosure does not need to constantly refresh and render all materials during the target operation, but only needs to render the materials other than the target material once when the dragging starts, and perform the full-screen rendering operation again at the end of the dragging. During the target operation, it is not affected by image effect processing such as special effects, algorithms, etc., but only needs to adjust the display state of the first canvas component corresponding to the target material, so as to better reduce the rendering time, effectively improve the visual fluency during the target operation, and protect the user's visual experience.

[0056] The above method provided by the embodiments of the present disclosure further includes: deleting the first canvas component after the target material is rendered. The first canvas component can be deleted immediately after the target material is rendered, or the first canvas component can be deleted after the full-screen rendering refreshing the canvas. The specific setting can be flexible, and compared with the method of deleting the first canvas component first and then rendering the target material, the above method can effectively avoid the blank delay phenomenon caused by deleting the first canvas component first and the target material not being rendered to the canvas in time, and effectively protect the visual experience.

[0057] In the step of displaying the synthesized image of the materials other than the target material on the second canvas component in the step S1102, the target material can be hidden, and the synthesized image of the materials other than the target material is rendered and displayed on the second canvas component. By hiding the target material first, the materials other than the target material can be quickly and reliably rendered. The embodiments of the present disclosure also provide two ways to hide the target material:

[0058] The first way is to change the current coordinate of the target material to a coordinate outside the second canvas component. It can also be understood as changing the current coordinate of the target material to a coordinate outside the draft canvas. By changing the coordinate position of the target material, the target material after position change no longer appears on the current canvas.

[0059] The second way is to set the target material to be fully transparent. That is, the transparency of the target material is set to 0, so that the target material achieves the hiding effect of full transparency.

[0060] In actual application, the first way or the second way can be selected flexibly according to the needs, which is not limited here.

[0061] In the step S1106, based on the action of changing the display state of the target material in the operation process of the target operation, the display state of the first canvas component on the second canvas component can be adjusted according to the following steps A to C:

[0062] Step A, obtaining the operation type of the target operation; wherein the operation type includes a drag operation, a rotation operation or a scaling operation.

[0063] Step B, determining the target key point of the first canvas component based on the operation type of the target operation.

[0064] The number of target key points is one or more. For ease of processing, the target key point can be a specific key point. Different operation types correspond to different target key points. For example, when the operation type of the target operation is a drag operation, the target key point of the first canvas component is determined to be a specified vertex of the first canvas component, such as the top left corner vertex of the first canvas component; when the operation type of the target operation is a rotation operation or a scaling operation, the target key point of the first canvas component is determined to be the center point of the first canvas component.

[0065] In step C, during the target operation, the display state changing action of the target material in the target operation is obtained at a preset time interval, and based on the display state changing action of the target material in the target operation, the target coordinates of the target key points of the first canvas component on the second canvas component are determined, and the position and / or size of the first canvas component on the second canvas component are adjusted according to the target coordinates. The above-mentioned time interval can be flexibly set according to requirements, such as being set according to a desired frame rate, and the shorter the time interval, the better the visual fluency. In addition, the target operation can be a drag operation, a rotation operation or a zoom operation, and the display state changing action of the target material in the target operation can be obtained by related technologies, such as through finger touch track or tool operation information, which will not be described here. In this way, the position coordinates or size of the first canvas component can be adjusted in real time, and the user is presented with the display state changing process of the target material during the target operation.

[0066] To facilitate understanding of the multimedia processing method provided by the embodiments of the present disclosure, the three track axes shown in FIG. 7 are still taken as an example for illustrative description, and the drag implementation process can be performed according to the following steps 1-6:

[0067] In step 1, before the drag starts, the rendering schematic diagram shown in FIG. 8 can be referred to.

[0068] In step 2, after detecting that the drag operation on the material 1 starts, the material 1 is hidden, and the current canvas is rendered. In actual applications, the material 1 can be hidden by changing the coordinates of the material 1 to outside the canvas, that is, the material 1 does not appear on the rendered canvas, and the transparency of the material can also be set to 0. For example, referring to the rendering effect schematic diagram shown in FIG. 12, the material 1 is hidden by changing the coordinates of the material 1 to outside the canvas, and special effects / algorithms and other materials except the material 1, such as the material 2, are rendered; wherein the material 2 is not affected by the special effects / algorithms.

[0069] In step 3, the RGBA image corresponding to the material 1 at the current time is obtained.

[0070] In step 4, a Canvas Dom (a web canvas component, which corresponds to the first canvas component described above) is created, and the RGBA image corresponding to the material 1 (that is, the current image of the material 1) is loaded into the Canvas Dom. For ease of understanding, refer to the canvas component schematic diagram shown in FIG. 13, which shows a Canvas Dom schematic diagram loaded with the current time image of the material 1 (that is, the RGBA image of the material 1).

[0071] Step 5, the Canvas Dom is placed on the rendering canvas Dom (corresponding to the aforementioned second canvas component). During the dragging process, only the coordinates of the Canvas Dom need to be changed according to the dragging trajectory, and there is no need to perform the operation of refreshing the rendering canvas and on-screen rendering. For ease of understanding, the Canvas Dom placed on the rendering canvas Dom is schematically shown in the canvas component diagram shown in FIG. 14. In actual application, since the Canvas Dom is independent of the rendering canvas Dom, it is equivalent to that the rendering canvas is not changed, and only the coordinate position of the Canvas Dom corresponding to the material 1 is changed, so the material 1 does not need to superimpose special effects / algorithms during the dragging process.

[0072] Step 6, the dragging operation is ended. At this time, the coordinates of the material 1 can be restored to the target coordinates (x2, y2) after the dragging, which can be determined based on the key point coordinates of the Canvas Dom at the end of the dragging operation, and then the operation of full on-screen rendering and refreshing the canvas is re-executed, wherein the material 1 layer is superimposed with special effects / algorithms. Then the Canvas Dom corresponding to the material 1 can be deleted. In this way, the problem that the material 1 cannot be refreshed to the canvas in time due to the deletion of the Canvas Dom in advance can be effectively avoided, and the blank delay of the material 1 can be avoided. At this time, the effect diagram after full on-screen rendering can be referred to FIG. 15.

[0073] In summary, the multimedia processing method provided by the embodiments of the present disclosure can be well applied to multimedia editing scenes such as web video editors, and when the user performs operations such as dragging, selecting, and zooming, the performance of single-frame preview rendering can be effectively optimized. By hiding the target material, creating and adjusting the display state of the canvas component corresponding to the target material during the operation process, etc., the current rendering canvas will not be affected during the operation process, and the current rendering canvas does not need to be full on-screen refreshed during the operation process, which can effectively save time and resources. No matter how large the magnitude of the specified algorithm such as global special effect is, it will not affect the performance during the target operation process, such as stable dragging performance for any image, video, or other materials. In summary, compared with related technologies, the embodiments of the present disclosure greatly improve the visual fluency of the target operation process, effectively guaranteeing the user experience.

[0074] Corresponding to the aforementioned multimedia processing method, FIG. 16 is a structural schematic diagram of a multimedia processing device provided by an embodiment of the present disclosure. The device can be realized by software and / or hardware, and can be integrated in an electronic device. As shown in FIG. 16, the multimedia processing device includes:

[0075] The component display module 1602 is configured to, in response to a start of a target operation on a target material in a multimedia editing window, display an image of the target material on a first canvas component, and display a composite image of materials other than the target material on a second canvas component, wherein the target material is a material on which an image effect processing is performed, the target operation is an operation of changing a display state of the target material, and the materials other than the target material are materials other than the target material in all to-be-rendered materials corresponding to a target preview position.

[0076] The component adjustment module 1602 is configured to place the first canvas component on the second canvas component, and adjust a display state of the first canvas component on the second canvas component based on an action of changing the display state of the target material in a process of the target operation.

[0077] The rendering module 1606 is configured to, in response to an end of the target operation, render the all to-be-rendered materials based on a display state of the first canvas component corresponding to the end of the target operation and a layer level of the all to-be-rendered materials, to obtain a preview image.

[0078] Compared with the related art, the above device provided in the embodiments of the present disclosure does not need to constantly refresh and render all materials during the target operation, and is not affected by image effect processing such as special effects, algorithms, and the like during the target operation, and only needs to adjust the display state of the first canvas component corresponding to the target material, thereby reducing the rendering time consumption, improving the visual fluency during the target operation, and ensuring the user experience.

[0079] In some embodiments, the device further includes a component deletion module configured to delete the first canvas component after the target material is rendered.

[0080] In some embodiments, the component display module 1602 is specifically configured to hide the target material, and render a composite image of the materials other than the target material, and display the composite image on the second canvas component.

[0081] In some embodiments, the component display module 1602 is specifically configured to change a current coordinate of the target material to a coordinate outside the second canvas component, or set the target material to be fully transparent.

[0082] In some embodiments, the component adjustment module 1604 is specifically configured to: obtain an operation type of the target operation; wherein the operation type comprises a drag operation, a rotation operation or a zoom operation; determine a target key point of the first canvas component based on the operation type of the target operation; obtain an action of changing a display state of the target material in the target operation at a preset time interval during operation of the target operation, and determine a target coordinate of the target key point of the first canvas component on the second canvas component based on the action of changing the display state of the target material in the target operation, and adjust the position and / or size of the first canvas component on the second canvas component according to the target coordinate.

[0083] In some embodiments, the component adjustment module 1604 is specifically configured to: in a case where the operation type of the target operation is a drag operation, determine that the target key point of the first canvas component is a specified vertex of the first canvas component; and in a case where the operation type of the target operation is a rotation operation or a zoom operation, determine that the target key point of the first canvas component is a center point of the first canvas component.

[0084] In some embodiments, the display state of the rendered target material on the preview image is consistent with a corresponding display state of the first canvas component at the end of the target operation; wherein the display state is characterized by position and / or size.

[0085] The multimedia processing apparatus provided by the embodiments of the present disclosure can perform the multimedia processing method provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device embodiments described above can refer to the corresponding process in the method embodiments, which will not be repeated here.

[0087] The embodiments of the present disclosure provide an electronic device, which comprises: a storage device having a computer program stored thereon; and a processing device configured to execute the computer program in the storage device to implement the steps of any method of the present disclosure.

[0088] Reference is now made to FIG. 17, which shows a structural diagram of an electronic device 1700 suitable for use in implementing embodiments of the present disclosure. The terminal device in embodiments of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (e.g., a car navigation terminal), and the like, as well as a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in FIG. 17 is merely an example and should not impose any limitation on the functions and use range of embodiments of the present disclosure.

[0089] As shown in FIG. 17, the electronic device 1700 can include a processing device (e.g., a central processor, a graphic processor, etc.) 1701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1702 or a program loaded from a storage device 1708 into a random access memory (RAM) 1703. In the RAM 1703, various programs and data required for the operation of the electronic device 1700 are also stored. The processing device 1701, the ROM 1702, and the RAM 1703 are connected to each other through a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0090] In general, the following devices can be connected to the I / O interface 1705: input devices 1706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 1707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 1708 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 1709. The communication devices 1709 can allow the electronic device 1700 to communicate with other devices wirelessly or via wires to exchange data. Although FIG. 17 shows the electronic device 1700 having various devices, it should be understood that all of the illustrated devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.

[0091] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 1709, or installed from the storage devices 1708, or installed from the ROM 1702. When the computer program is executed by the processing device 1701, the above-described functions defined in the methods of embodiments of the present disclosure are performed.

[0092] In addition to the method and device described above, the embodiments of the present disclosure can also be a computer program product, which includes computer program instructions that make the processor execute the image processing method provided by the embodiments of the present disclosure when the processor runs. The computer program product can be written in any combination of one or more programming languages to execute the program code of the embodiments of the present disclosure, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0093] In addition, the embodiments of the present disclosure can also be a computer readable storage medium, which stores computer program instructions, and the computer program instructions make the processor execute the multimedia processing method provided by the embodiments of the present disclosure when the processor runs.

[0094] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of readable storage medium include: electrical connection with one or more conductive wires, portable disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0095] The embodiments of the present disclosure also provide a computer program product, which includes computer programs / instructions that are executed by a processor to implement the multimedia processing method in the embodiments of the present disclosure.

[0096] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained according to relevant laws and regulations.

[0097] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed by the user will need to acquire and use personal information of the user. Thus, the user can autonomously select whether to provide the personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium performing the operation of the technical solution of the present disclosure according to the prompt information.

[0098] As an optional but non-limiting implementation, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, and the prompt information may be presented in the pop-up window in the form of text. In addition, the pop-up window may also carry a selection control for the user to select “agree” or “disagree” to provide personal information to the electronic device.

[0099] It can be understood that the above notification and acquisition of user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0100] It should be noted that in this document, relational terms such as “first” and “second”, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element preceded by “comprises a...” does not, without more limitations, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the recited element.

[0101] The above description is merely one specific implementation of the present disclosure, which enables those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimedia processing method, comprising: in response to a start of a target operation on a target material in a multimedia editing window, displaying an image of the target material on a first canvas component and displaying a composite image of materials other than the target material on a second canvas component; wherein the target material is a material of an editing operation to which an image effect processing is applied, the target operation is an operation to change a display state of the target material, and the materials other than the target material are materials other than the target material among all to-be-rendered materials corresponding to a target preview position; placing the first canvas component on the second canvas component and adjusting a display state of the first canvas component on the second canvas component based on an action of changing the display state of the target material during an operation process of the target operation; in response to an end of the target operation, rendering the all to-be-rendered materials based on a display state of the first canvas component corresponding to the end of the target operation and layer levels of the all to-be-rendered materials to obtain a preview image. 2.The method of claim 1, further comprising: deleting the first canvas component after the target material is rendered. 3.The method of claim 1, wherein the displaying the composite image of the materials other than the target material on the second canvas component comprises: hiding the target material and rendering a composite image of the materials other than the target material to display the composite image on the second canvas component. 4.The method of claim 3, wherein the hiding the target material comprises: changing a current coordinate of the target material to a coordinate outside the second canvas component; or setting the target material to be fully transparent. 5.The method of claim 1, wherein the adjusting the display state of the first canvas component on the second canvas component based on the action of changing the display state of the target material during the operation process of the target operation comprises: obtaining an operation type of the target operation; wherein the operation type comprises a drag operation, a rotation operation or a zoom operation; determining a target key point of the first canvas component based on the operation type of the target operation; during the operation of the target operation, obtaining the action of changing the display state of the target material at a preset time interval and determining a target coordinate of the target key point of the first canvas component on the second canvas component based on the action of changing the display state of the target material, and adjusting a position and / or size of the first canvas component on the second canvas component according to the target coordinate. 6.The method of claim 5, wherein the determining the target key point of the first canvas component based on the operation type of the target operation comprises: in a case where the operation type of the target operation is the drag operation, determining the target key point of the first canvas component to be a specified vertex of the first canvas component. ​ In a case where the operation type of the target operation is a rotation operation or a zoom operation, the target key point of the first canvas component is determined as a center point of the first canvas component.

7. The method of claim 1, wherein the display state of the rendered target material on the preview image is consistent with the display state of the first canvas component corresponding to the end of the target operation; wherein, The display state is characterized by a position and / or a size. 8.A multimedia processing apparatus, comprising: a component display module configured to, in response to a start of a target operation on a target material in a multimedia editing window, display an image of the target material on a first canvas component and display a composite image of materials other than the target material on a second canvas component, wherein the target material is a material on which an image effect processing is applied by an editing operation, the target operation is an operation of changing a display state of the target material, and the materials other than the target material are materials other than the target material among all to-be-rendered materials corresponding to a target preview position; a component adjustment module configured to place the first canvas component on the second canvas component and adjust a display state of the first canvas component on the second canvas component based on an action of changing the display state of the target material in an operation process of the target operation; a rendering module configured to, in response to an end of the target operation, render the all to-be-rendered materials based on a display state of the first canvas component corresponding to the end of the target operation and layer levels of the all to-be-rendered materials, to obtain a preview image. 9.An electronic device, comprising: a storage device having a computer program stored thereon; a processing device configured to execute the computer program in the storage device to implement steps of the multimedia processing method according to any one of claims 1-7. 10.A computer readable storage medium, having a computer program stored thereon, the computer program being configured to implement the multimedia processing method according to any one of claims 1-7. 11.A computer program product, comprising a computer program, the computer program being configured to implement the multimedia processing method according to any one of claims 1-7 when executed by a processor.

Images