Multimedia processing method, system, and vehicle

Abstract

The embodiment of the present application provides a kind of multimedia processing method, system and vehicle, the multimedia processing method includes: obtaining the image matched with music file;In response to the adjustment instruction for image, at least one image parameter of image is adjusted;According to the image parameter of adjustment, the music parameter corresponding to image parameter is adjusted;According to the music parameter of adjustment, music file is updated.Tuning function based on image adjustment can be realized, and user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of multimedia, and in particular to a multimedia processing method, system, and vehicle. Background Technology

[0002] With the improvement of people's living standards and the increase in entertainment demand, technologies such as multimedia processing and multimedia playback have developed rapidly, and the demand for various functions of multimedia devices has also been growing.

[0003] Music is an important form of multimedia, occupying a significant place in people's daily lives and cultural exchanges. Currently, mainstream music playback software, in addition to basic functions such as playing music and adjusting volume, also supports the simultaneous display of lyrics, artists, album art, and the selection of sound effect modes, such as pop, rock, classical, jazz, and bass boost. Some software also allows users to customize sound effect parameters, for example, by adjusting the gain of different frequencies through tools such as equalizers (EQ) to create personalized sound quality.

[0004] It should be noted that the above description of the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of the present invention and facilitating understanding by those skilled in the art. It should not be assumed that the above technical solutions are known to those skilled in the art simply because they have been described in the background section of this invention. Summary of the Invention

[0005] The inventors discovered that in the aforementioned prior art, adjusting various sound effect parameters through tools such as equalizers requires a certain level of expertise from the user, thus limiting the user base; in addition, the sound effect parameter adjustment interface of tools such as equalizers is relatively monotonous and lacks intuitive feedback for adjustment, resulting in a poor user experience.

[0006] In addition, during music playback, the content displayed synchronously is usually lyrics or simple and fixed images, which cannot fully showcase the characteristics or connotations of the currently playing music.

[0007] To address one or more problems or similar issues in the prior art, embodiments of the present invention provide a multimedia processing method, system, and vehicle that can achieve image-based audio tuning functions and improve user experience.

[0008] According to a first aspect of the present invention, a multimedia processing method is provided, the method comprising: acquiring an image matching a music file; adjusting at least one image parameter of the image in response to an adjustment instruction for the image; adjusting music parameters corresponding to the adjusted image parameters according to the adjusted image parameters; and updating the music file according to the adjusted music parameters. By utilizing the association between image parameters and music parameters, a user can adjust the associated music parameters by adjusting the image parameters of an image matching music, thereby enabling music modification, i.e., achieving image-based tuning functionality. Therefore, users can achieve tuning functionality in an intuitive, simple, and novel way, enhancing usability and user experience.

[0009] Optionally, the method further includes: generating adjustment instructions for the image based on the user's voice command, and / or generating adjustment instructions for the image based on the user's adjustment operations on image parameters. This allows image adjustment instructions to be generated through multiple methods, such as user voice or user operation, making it more convenient for users and further improving usability.

[0010] Optionally, the voice command is an adjustment command for image parameters, and / or the voice command expresses the user's requirements or expectations for the adjustment effect. In this way, users can express voice commands for image or music adjustments in multiple ways. Even without explicit parameter adjustment commands, adjustments can still be achieved as long as the desired effect is expressed. For example, the user can simply state the desired adjustment effect, further enhancing user convenience and experience.

[0011] Optionally, the system can receive voice commands from the user regarding their desired adjustment effects, generate recommended adjustment schemes for the image parameters, and generate adjustment instructions for the image after the user confirms or modifies them. This way, even if the user doesn't provide specific adjustment settings, recommended adjustment schemes can be provided based on the user's voice commands for confirmation or modification. The confirmed schemes will better meet the user's needs, further enhancing the user experience.

[0012] Optionally, the method further includes: displaying a control for at least one of the image parameters; and having a user operate on the displayed control for the image parameters to generate the adjustment instructions for the image. In this way, adjusting the corresponding image parameters based on the displayed controls allows for convenient adjustment of the image parameters.

[0013] Optionally, the type and / or number of controls for the displayed image parameters can be set and / or adjusted by the user. This allows users to customize the image parameter controls according to their needs or preferences, further enhancing the user experience.

[0014] Optionally, the music parameters associated with the adjusted image parameters are adjusted, including: adjusting the weights of the music parameters associated with the image parameters based on changes in the weights of the adjusted image parameters. This enables precise and rapid adjustment of the corresponding music parameters.

[0015] Optionally, the correspondence between the image parameters and the music parameters is preset, and / or the correspondence can be set and / or adjusted by the user. This allows users to set or adjust the correspondence between image parameters and music parameters according to their usage habits or preferences, further enhancing the user experience.

[0016] Optionally, the method further includes displaying explanatory information about the relationship between the image parameters and the music parameters. This facilitates user understanding and use of the image parameter controls to adjust the music.

[0017] Optionally, the method further includes: updating the image according to the adjusted image parameters; playing the music corresponding to the updated music file and simultaneously displaying the updated image. In this way, playing the updated music and simultaneously displaying the updated image allows the user to intuitively experience the effect of the adjustment from both auditory and visual perspectives.

[0018] Optionally, the image parameters include at least one of the following: image brightness, contrast, saturation, image style, color, size, subject content, and dynamic effects; and / or, the music parameters include at least one of the following: music pitch, timbre, tone, volume, music style, reverb, delay, gain, music stage, crossover sound pressure level, amplifier channel, and rhythm. In this way, by associating multiple dimensions of image parameters with multiple dimensions of music parameters, flexible and comprehensive adjustment of both the music and the matching image can be achieved.

[0019] Optionally, obtaining an image matching a music file includes: obtaining the music file; generating music features of the music file based on the music file; determining an image theme and visual template matching the music features; determining image features associated with the music features; and generating an image matching the music file based on the image theme, the visual template, and the image features.

[0020] In this way, by using image themes and visual templates that match the music features of the music file, as well as image features associated with the music features, a highly matching, vivid, and unique image is generated. This allows the matching image to be displayed synchronously while the music is playing. Furthermore, because the image is generated by using image themes and visual templates that match the music features, combined with image features associated with the music features, the generated image reflects the characteristics of the music from multiple dimensions. This enriches the image's presentation, content, and details. When the generated image is displayed while the music is playing, users can intuitively experience the characteristics and connotations of the music from both visual and auditory perspectives, easily evoking resonance and effectively enhancing the user experience.

[0021] Optionally, the method further includes: during the playback of the music corresponding to the music file, the music features are dynamically updated in real time, and the image is dynamically updated according to the dynamically updated music features. In this way, dynamically updating the image during music playback presents the user with images dynamically related to the song's progression, climax, and expressed emotions, allowing the user to simultaneously experience the changes in both the song and the image, further enhancing the user experience.

[0022] Optionally, during the playback of the music corresponding to the music file, the music features are dynamically updated in real time, and the image is dynamically updated based on the dynamically updated music features. This includes: dynamically updating the music features according to at least one of the music's playback duration, structure, rhythm, and emotional changes; and dynamically updating at least one of the image's display parameters, display content, and display format according to the dynamically updated music parameters. In this way, based on the music's progression, climax, and emotional changes, the image's display parameters such as brightness and saturation, or the displayed content, or the display format (e.g., dynamic effects) are dynamically updated, further enhancing the display effect of the image changing synchronously with the music.

[0023] Optionally, the method further includes: displaying music tags corresponding to the music features; and dynamically updating the music features in real time during the playback of the music corresponding to the music file, and dynamically updating the music tags according to the dynamically updated music features. In this way, by dynamically updating the music tags according to the music playback, users can intuitively perceive the changes in music features, further enhancing the user experience.

[0024] Optionally, the method further includes: adjusting the music tag in response to an adjustment instruction for the music tag; adjusting the music features corresponding to the music tag according to the adjusted music tag; and updating the music file according to the adjusted music features. This enables a tuning function based on music tag adjustments, improving the user experience.

[0025] Optionally, the method further includes: generating and displaying recommended adjustment settings related to images and / or music based on the analysis results of the collected user data. This enables the provision of recommendations for adjustment settings based on popularity and trends, leveraging big data analysis to help users quickly find settings that suit their preferences and further enhance the user experience.

[0026] According to a second aspect of the present invention, a multimedia system is provided, comprising a multimedia processing apparatus for implementing the multimedia processing method of the first aspect of the present invention, the system further comprising: a sound output device that outputs music corresponding to a music file; and a display device that synchronously displays an image matching the music file when outputting the music corresponding to the music file.

[0027] According to a third aspect of the present invention, a vehicle is provided, the vehicle including the multimedia system described in the second aspect of the present invention.

[0028] One of the beneficial effects of the embodiments of the present invention is that:

[0029] By leveraging the correlation between image parameters and music parameters, users can adjust the image parameters of an image that matches the music, thereby adjusting the associated music parameters accordingly. This enables music modification, achieving image-based tuning functionality. As a result, users can achieve tuning in an intuitive, simple, and novel way, enhancing the fun and ease of use and improving the user experience.

[0030] Embodiments of the present invention are disclosed in detail with reference to the following description and accompanying drawings. It should be understood that the embodiments of the present invention are not limited in scope thereto. Within the spirit and scope of the appended claims, embodiments of the present invention include many changes, modifications, and equivalents.

[0031] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0032] It should be emphasized that the term "including / contains / has" as used herein refers to the presence of a feature, whole or component, but does not exclude the presence or addition of one or more other features, wholes or components. Attached Figure Description

[0033] The above and other objects, features and advantages of embodiments of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0034] Figure 1This is a block diagram of a multimedia processing method according to an embodiment of the present invention;

[0035] Figure 2 This is a block diagram of a music playback interface according to an embodiment of the present invention;

[0036] Figure 3 This is another block diagram of the music playback interface according to an embodiment of the present invention;

[0037] Figure 4 This is a block diagram of a method for implementing step 101 in an embodiment of the present invention;

[0038] Figure 5 This is another block diagram of the multimedia processing method according to an embodiment of the present invention;

[0039] Figure 6 This is another block diagram of the multimedia processing method according to an embodiment of the present invention;

[0040] Figure 7 This is another block diagram of the multimedia processing method according to an embodiment of the present invention;

[0041] Figure 8 This is a block diagram of a multimedia processing device according to an embodiment of the present invention;

[0042] Figure 9 This is a block diagram of a multimedia system according to an embodiment of the present invention. Detailed Implementation

[0043] Referring to the accompanying drawings, the foregoing and other features of the invention will become apparent from the following description. Specific embodiments of the invention are specifically disclosed in the description and drawings, illustrating partial implementations in which the principles of the invention can be employed. It should be understood that the invention is not limited to the described embodiments, but includes all modifications and equivalents falling within the scope of the appended claims.

[0044] The multimedia processing method, apparatus, system, and vehicle of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] First aspect of the embodiments

[0046] This invention provides a multimedia processing method.

[0047] The multimedia processing method of this invention can be executed by one or more devices or equipment with multimedia processing capabilities. For example, all steps of the method can be executed by the same device or equipment, or the various steps of the method can be executed separately by different devices or equipment. This invention does not limit the type or number of entities executing the method.

[0048] Furthermore, the apparatus or device for executing the multimedia processing method of this invention can be a local device or a cloud-based device. For example, local devices include, but are not limited to, multimedia playback devices, mobile terminals, and vehicle-mounted terminals, while cloud-based devices include, but are not limited to, cloud servers. Alternatively, it can be a collection of the aforementioned different devices, i.e., different devices perform a portion of the steps respectively.

[0049] Figure 1 This is a block diagram of a multimedia processing method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0050] Step 101: Obtain the image that matches the music file;

[0051] Step 102: In response to an adjustment instruction for the image, adjust at least one image parameter of the image;

[0052] Step 103: Adjust the music parameters corresponding to the adjusted image parameters;

[0053] Step 104: Update the music file according to the adjusted music parameters.

[0054] In this way, by leveraging the correlation between image parameters and music parameters, users can adjust the associated music parameters by adjusting the image parameters of the image that matches the music, thereby modifying the music and achieving image-based tuning. As a result, users can achieve tuning in an intuitive, simple, and novel way, enhancing the fun and ease of use and improving the user experience.

[0055] The steps of the method are described in detail below. This description is intended to provide a better understanding of the invention, not to limit it.

[0056] In step 101, an image matching the music file is obtained. The specific method for obtaining this image will be explained in detail later.

[0057] After obtaining the image that matches the music file, the image can be displayed, and in step 102, in response to the adjustment instruction for the image, at least one image parameter of the image is adjusted.

[0058] In some embodiments, image parameters can be various parameters related to the image, such as at least one of the image's brightness, contrast, saturation, image style, color, size, subject content, and dynamic effects.

[0059] In some embodiments, the types of image parameters that can be adjusted can be preset, or they can be set or selected by the user.

[0060] In some embodiments, in step 102, adjustment instructions for the image can be generated based on the user's voice commands; alternatively, adjustment instructions can also be generated based on the user's adjustments to image parameters. This allows for the generation of image adjustment instructions through various methods, such as user voice or user operation, making it more convenient for users and further improving usability.

[0061] In cases based on user voice commands, the user's voice commands can be recognized and converted into text, and adjustment instructions for the image can be generated based on the text.

[0062] In some embodiments, voice commands are instructions for adjusting image parameters, and / or, voice commands express the user's requirements or expectations for the adjustment effect. This allows users to express voice commands for image or music adjustments in various ways; for example, the user can simply state the desired adjustment effect, further enhancing user convenience and experience. For instance, if a user issues the voice command "reduce the image brightness," after voice recognition, a pre-set reduction amount (e.g., 10%) can be combined to generate the adjustment command "reduce the image brightness by 10%"; alternatively, the system can interact with the user via voice, asking for the specific reduction amount, and generating the corresponding adjustment command after the user's response.

[0063] In some embodiments, the system receives voice commands from the user regarding their desired adjustment effect, generates recommended adjustment schemes for the image parameters, and generates adjustment instructions for the image after the user confirms or modifies them. This way, even if the user does not provide specific adjustment settings, recommended adjustment schemes can be provided based on the user's voice commands for confirmation or modification. The confirmed schemes are more closely aligned with the user's needs, further enhancing the user experience.

[0064] For example, if a user issues a voice command "I want to be sad", the system receives and parses the user's voice command, and generates an "image adjustment scheme and / or music adjustment scheme" that matches the keyword "sad". In addition, the system can confirm the generated "image adjustment scheme and / or music adjustment scheme" with the user through voice interaction, and implement it after the user confirms.

[0065] For adjustment-based operations, controls for at least one image parameter can be displayed. Users can generate adjustment commands for the image by interacting with these controls. This allows users to easily adjust image parameters by using the displayed controls.

[0066] In some embodiments, the user's adjustment of image parameters includes: the user manipulating the displayed image parameter controls to adjust the weights of the image parameters. Correspondingly, in step 103, the weights of the music parameters associated with the image parameters are adjusted based on the change in the adjusted image parameter weights, so that in step 104, the music file is updated based on the adjusted music parameters. For example, the user can drag the image parameter controls to adjust the weights of the image parameters. For example, the user drags the control corresponding to "brightness" from the value "10%" to "20%".

[0067] For example, the weight of an image parameter refers to the degree of influence that image parameter has on the image, while the weight of a music parameter refers to the degree of influence that music parameter has on the music.

[0068] In this way, by manipulating the controls on the displayed image parameters and adjusting the weight changes of the image parameters to adjust the weights of the associated music parameters, the corresponding adjustments of the music parameters can be achieved accurately and quickly.

[0069] Alternatively, the specific values ​​of the image and music parameters can be adjusted directly. For example, adjusting image parameters involves the user manipulating the displayed image parameter controls to adjust the specific values ​​of the image parameters. Correspondingly, in step 103, based on the change in the adjusted image parameter values, the specific values ​​of the music parameters associated with the image parameters are adjusted, so that in step 104, the music file is updated based on the adjusted music parameters. For example, the user can drag the image parameter controls to adjust the values ​​of the image parameters. For example, the user can drag the control corresponding to "brightness" from the value "155" to "170".

[0070] In some embodiments, the type and / or number of controls for the displayed image parameters can be set and / or adjusted by the user. This allows users to customize the controls for the displayed image parameters according to their needs or preferences, further enhancing the user experience.

[0071] In some embodiments, the correspondence between image parameters and music parameters is preset, and / or the correspondence can be set and / or adjusted by the user. This allows users to set or adjust the correspondence between image parameters and music parameters according to their usage habits or preferences, further enhancing the user experience.

[0072] In some embodiments, music parameters can be various music-related parameters. For example, music parameters can be represented by an equalizer; changes in music parameters can be reflected as changes in the equalizer curve. Of course, other representation methods are also possible.

[0073] For example, music parameters include at least one of the following: pitch, timbre, tone, volume, music style, reverberation, delay, gain, music stage, crossover sound pressure level, amplifier channel, and rhythm.

[0074] In this way, by using image parameters of multiple dimensions, music parameters of multiple dimensions can be associated, enabling flexible and comprehensive adjustment of both the music and the matching image.

[0075] In some embodiments, artificial intelligence techniques can be used to update music files. For example, a style transfer model can be used to apply adjustments to music parameters to different types of music. Another example is the use of multimodal alignment techniques to synchronize image parameter adjustments with music parameter adjustments.

[0076] In some embodiments, controls for multiple image parameters may be displayed overlaid on the displayed image, or controls for multiple image parameters may be displayed around the image. For example, controls for adjustable image parameters may be displayed on one or both sides of the image, or controls for adjustable image parameters may be displayed around the image. This embodiment of the invention does not limit the number or layout of the displayed image parameter controls.

[0077] Users can adjust multiple image parameters through various operations, such as sliding, scrolling, and clicking, to adjust the image parameters, such as adjusting the weight of the image parameters.

[0078] In this way, by adjusting the weights of the corresponding image parameters based on the displayed controls, image parameters can be easily adjusted.

[0079] In step 104, the music file is updated based on the adjusted music parameters. For example, changes in music parameters can be reflected in changes to the equalizer curve. The audio signal of the music is processed based on the adjusted equalizer curve to obtain the updated music file.

[0080] Additionally, when playing updated music files, it can be assumed that the processed audio signal is being played. Of course, other methods are also possible.

[0081] Figure 2 This is a block diagram of a music playback interface according to an embodiment of the present invention. Figure 2 As shown, at the top of the music playback interface 200, an image 201 matching the music is displayed. Below the image 201, the music name 202, artist 203, and playback controls 204 are displayed. Furthermore, multiple image parameter controls 205 are set on both sides of the image. Figure 2 The image shows six image parameter controls 205-1, 205-2, ... 205-6 as an example.

[0082] in addition, Figure 2 The music playback interface shown is just an example. The number of image parameter controls can be determined according to the actual situation. In addition, other layout methods can be used, and other content can be displayed, such as lyrics, other controls, etc.

[0083] In some embodiments, after clicking on an image parameter control, an adjustment slider for that image parameter pops up. By dragging the slider, the weight or value of the image parameter can be adjusted.

[0084] For example, the slider can slide horizontally, that is, adjust the image parameters by dragging the slider horizontally; or the slider can slide vertically, that is, adjust the image parameters by dragging the slider up and down.

[0085] Alternatively, the image parameter control can be designed to allow users to slide up and down or left and right to adjust the weight or value of the image parameter.

[0086] In some embodiments, image parameters and music parameters are pre-associated. The rules for this association can be preset or set or adjusted by the user. That is, image parameters and music parameters correspond, and the correspondence is preset and / or can be set and / or adjusted by the user.

[0087] For example, Table 1 shows the correspondence between image parameters and music parameters. However, this is just an example; the two parameters can be corresponded in other ways. In addition, some of the image parameters and music parameters in Table 1 can be used, or other image parameters and music parameters not shown in Table 1 can be included.

[0088] Table 1. Correspondence between image parameters and music parameters

[0089] Serial Number Image parameters Music parameters 1 brightness pitch 2 Saturation timbre 3 Softness tone 4 Contrast Delay 5 main body reverberation 6 Dynamic effects Rhythm 7 Image style Musical Style 8 Theme Image Style EQ style …… …… ……

[0090] For example, Figure 2 Image parameter 1 represents the brightness of the image, and its associated musical parameter is pitch; image parameter 2 represents the saturation of the image, and its associated musical parameter is timbre; image parameter 3 represents the smoothness of the image, and its associated musical parameter is tone; image parameter 4 represents the contrast of the image, and its associated musical parameter is delay; image parameter 5 represents the subject of the image, and its associated musical parameter is reverb; image parameter 6 represents the dynamic effect of the image, and its associated musical parameter is rhythm.

[0091] In some embodiments, when the weight or value of an image parameter is increased, the weight or value of the music parameter associated with that image parameter is also increased accordingly; when the weight or value of an image parameter is decreased, the weight or value of the music parameter associated with that image parameter is also decreased accordingly; the adjustment range of the music parameter and the image parameter can be the same, or a preset adjustment ratio between the two can be used.

[0092] In some embodiments, users can customize or adjust the type and number of image parameter controls in the music playback interface, thereby providing a personalized tuning module and further enhancing the user experience.

[0093] In some embodiments, explanatory information regarding the relationship between image parameters and music parameters can also be displayed. For example, this information can be displayed using a chart similar to Table 1, or by clicking on the corresponding image parameter control to bring up relevant explanatory information. This facilitates user understanding and use of the image parameter controls to adjust the music.

[0094] For example, by clicking Figure 2 The control 205-1 for image parameter 1 in the image parameter 1, or when a finger or mouse is placed on the position of image parameter 1, will pop up explanatory information such as "The pitch of the song can be adjusted by adjusting the image brightness"; in addition, it can also explain the adjustment method such as "You can adjust it by sliding your finger up and down".

[0095] In some embodiments, optionally, such as Figure 1 As shown, the method may further include:

[0096] Step 105: Update the image based on the adjusted image parameters;

[0097] Step 106: Play the music corresponding to the updated music file and display the updated image synchronously.

[0098] This allows users to intuitively experience the effects of the adjustment through both auditory and visual means.

[0099] For example, a user might want to adjust the reverb and pitch of the song "Summer Melancholy," such as lowering the reverb weight or value to the lowest level and increasing the pitch weight or value.

[0100] like Figure 2 As shown, by adjusting the image parameter control 205-1 that represents brightness, the user can increase the weight or value of brightness, thereby increasing the weight or value of pitch associated with brightness.

[0101] Additionally, by adjusting the image parameter control 205-5 representing the subject, the user can minimize the weight (e.g., the proportion of the subject in the entire image) or value (e.g., the size of the subject), thereby minimizing the weight or value of the reverberation associated with the subject.

[0102] Figure 3 This is another block diagram of the music playback interface according to an embodiment of the present invention. For example... Figure 3 As shown, after the above adjustments, the updated image 201' is generated and displayed, and the updated music is played simultaneously.

[0103] like Figure 3 As shown, the brightness of the image increases, and the main subject "little girl" in the original image 201 is deleted, resulting in the updated image 201'.

[0104] In this way, after making simple adjustments to the image parameter controls, users can intuitively and simultaneously feel the changes in the image and the music, bringing dual visual and auditory feedback. This adjustment method is novel and interesting, provides intuitive feedback, and easily gives a sense of accomplishment.

[0105] In addition, in some embodiments, if the user is not satisfied with the updated music and / or images, they can continue to adjust the image parameter controls until they are satisfied.

[0106] In addition, in some embodiments, voice adjustment can be combined with the adjustment of image parameter controls to further improve user convenience.

[0107] In some embodiments, the method further includes: generating and displaying recommended adjustment schemes related to images and / or music based on the analysis results of the collected user data.

[0108] In this way, based on big data analysis, recommendations for adjustment schemes can be provided based on popularity and trends, helping users quickly find adjustment schemes that suit their preferences and further improving the user experience.

[0109] The following is an exemplary implementation of the multimedia processing method described above, including the following processing.

[0110] 1. Provides a user-friendly adjustment interface, including:

[0111] Interface design: Create an intuitive and easy-to-use interface that allows users to easily adjust image parameters;

[0112] Visual elements: Use visual elements such as charts, sliders, and buttons to represent image parameters and their weights;

[0113] Labels and tips: Add clear labels and tips to each control to help users understand its function;

[0114] Interactive feedback: Ensure that users receive immediate visual feedback for every change they make to the interface, such as color changes or numerical updates;

[0115] Layout optimization: Based on user research and user experience testing results, optimize the interface layout to make it conform to user habits and intuition.

[0116] 2. Real-time feedback of adjustment results, including:

[0117] Audio processing: Receives user adjustments, applies them to the audio signal, and generates adjusted sound effects in real time;

[0118] Sound playback: Sends the adjusted audio signal stream to speakers or other output devices so that users can hear real-time feedback;

[0119] Visual feedback: Display the adjusted parameter weights in the user interface to intuitively understand the adjustment results;

[0120] Dynamic adjustment: As the user continues to adjust the image parameters, the audio and visual feedback is continuously updated to reflect the latest adjustment results.

[0121] 3. Provide recommendation adjustment schemes based on popularity and trends, including:

[0122] Data analytics: Collecting and analyzing data from large numbers of users, including their moderation plans and preferences;

[0123] Pattern recognition: Using data mining techniques to identify common adjustment patterns and trends;

[0124] Recommendation Algorithm: Based on the identified adjustment patterns and trends, develop a recommendation algorithm to provide users with personalized adjustment suggestions;

[0125] Recommended display: Display recommended adjustment schemes in the user interface to help users quickly find the adjustment scheme that suits their preferences;

[0126] User feedback: Collect user feedback on recommendation adjustment schemes to continuously improve the accuracy and effectiveness of recommendation algorithms.

[0127] The above provides a detailed description of the image-based tone adjustment function of this invention.

[0128] The following is a detailed explanation of how to obtain the image that matches the music file in step 101.

[0129] Figure 4This is a block diagram of a method for implementing step 101 of an embodiment of the present invention. For example... Figure 4 As shown, the method includes:

[0130] Step 401: Obtain the music file;

[0131] Step 402: Generate the musical characteristics of the music file based on the music file;

[0132] Step 403: Determine the image subject and visual template that match the music features;

[0133] Step 404: Determine the image features associated with the music features;

[0134] Step 405: Generate an image that matches the music file based on the image theme, visual template, and image features.

[0135] In this way, by using image themes and visual templates that match the music features of the music file, as well as image features associated with the music features, a highly matching, vivid, and unique image is generated. This allows the matching image to be displayed synchronously while the music is playing. Furthermore, because the image is generated by using image themes and visual templates that match the music features, combined with image features associated with the music features, the generated image reflects the characteristics of the music from multiple dimensions. This enriches the image's presentation, content, and details. When the generated image is displayed while the music is playing, users can intuitively experience the characteristics and connotations of the music from both visual and auditory perspectives, easily evoking resonance and effectively enhancing the user experience.

[0136] In step 401, music files can be obtained from a local device, a cloud server, or a user-created music file. This allows for various methods of obtaining music files, improving the flexibility of music file acquisition and the richness of music resources. The music file referred to here can be the audio signal corresponding to the music.

[0137] In some embodiments, the retrieval of music files can be triggered in various ways, such as retrieving the corresponding music file based on the user's selection, or retrieving the corresponding music file based on a push notification to the user.

[0138] In some embodiments, the music file may be a music file stored locally, a music file from the network, such as a music file on a cloud server, or a music file created by the user.

[0139] In some embodiments, a music file corresponds to a piece of music; that is, one music file corresponds to one piece of music, and the music file may include various files related to the music. For example, a music file may include at least one of the following: music title, artist information, lyrics, music release date, playback duration, and audio files. However, embodiments of the present invention do not limit the content contained in the music file.

[0140] After obtaining the music file, in step 402, the music characteristics of the music file are generated based on the music file.

[0141] In some embodiments, musical features may include at least one of musical style, lyrics, emotions or ideas expressed in the music, and musical rhythm.

[0142] For example, music styles include pop, rock, classical, jazz, electronic, folk, metal, and country music.

[0143] For example, the lyrics include delicate expressions of emotions such as love, friendship, and family affection; profound insights into dreams, growth, and life; observations and reflections on social phenomena and historical events; vivid depictions of natural landscapes and life scenes; and the display and inheritance of cultural traditions and the spirit of the times.

[0144] For example, music expresses a variety of emotions, including the sweet romance, longing and sorrow in love, the warmth, gratitude and remembrance in family, the joy of companionship and reluctance to part in friendship, personal emotions such as joy and excitement, loneliness, anger and rebellion, as well as the pursuit of ideals, praise of nature and reflection on life.

[0145] For example, the ideas expressed in music include the praise of love and beauty, the pursuit of freedom and dreams, the critique and reflection on social reality, the exploration of the meaning and value of life, the inheritance and remembrance of history and culture, and the contemplation on the relationship between humanity and nature.

[0146] For example, musical rhythms include steady and regular beat rhythms, rhythmic swing rhythms, impactful accented rhythms, and various distinctive rhythms formed by combinations of long and short notes.

[0147] In this way, by generating music features containing multiple dimensions of information based on the music file, the matching degree between the generated image and the music file can be further improved, and different aspects of the music can be displayed through the image. This allows users to enhance their auditory experience by visually perceiving the image while enjoying the music, thereby further improving the user experience.

[0148] In some embodiments, music features can be extracted directly from music files, for example, by using an artificial intelligence model; or, main music elements can be extracted from music files and music features can be generated based on these main music elements.

[0149] In some embodiments, the types of music elements are predetermined, for example, by identifying multiple candidate music elements based on a publicly available dataset, such as the MTT dataset. This allows for the comprehensive extraction of music elements.

[0150] In some embodiments, the type of musical element may include various sound attributes, sound types, pitches, and instruments, such as male voices, female voices, animal sounds, solos, harmonies, high notes, low notes, mid notes, guitars, pianos, drums, etc.

[0151] In some embodiments, the music file can be analyzed to obtain multiple music elements, and the correlation between the music file and the multiple music elements can be determined. At least one music element whose correlation meets a preset condition can be selected from the multiple music elements as the main music element belonging to the music file. For example, at least one music element whose correlation exceeds a preset threshold can be selected as the main music element of the music file.

[0152] In this way, by analyzing music files and identifying music elements that are highly relevant to the music files, these elements can be used as the main music elements in the generated images, thereby further improving the matching degree between the generated images and the music files and enhancing the user experience.

[0153] For example, a trained music classifier model can be used to extract key musical elements and generate musical features. For instance, the MusicNN model can be used to extract at least one key musical element belonging to a given piece of music. This could be achieved by selecting musical elements with higher weights or more prominent features as the key musical elements of the music file. Here, weights can represent relevance to the music. For example, brightness can be used to indicate the weight or prominence of each musical element; higher brightness indicates a higher weight or more prominent feature, meaning a stronger relevance to the music. A preset number of the brightest musical elements can be selected as the key musical elements of the music.

[0154] Alternatively, music features can be generated using the MusicNN model. For example, music features can be generated based on a predetermined number of selected music elements as the main music elements of a music file.

[0155] For example, if the main musical elements of a music file are identified as: male singer, guitar, strong rhythm, and drums, then the following musical characteristics are generated based on these main musical elements: rock music.

[0156] In addition, other information in the music file can be combined to generate musical features. For example, by combining the lyrics with the main musical elements mentioned above, namely male singer, guitar, strong rhythm, and drums, the following musical features can be generated based on these main musical elements: expressing individuality and freedom, and rock music.

[0157] In some embodiments, the type of music element can also be determined based on user selection or adjustment; that is, users can add or filter the music elements they want. This allows for the generation of images that are closer to user preferences, further enhancing the user experience.

[0158] In some embodiments, the types of music features are predetermined, and multiple candidate music features are determined based on publicly available datasets, such as the MTT dataset. This allows for the comprehensive extraction of music features.

[0159] In step 402, music features of the music file are generated based on the music file. The process may include the following: forming a music feature library based on a predetermined music feature type, analyzing and processing the audio signal of the music file to obtain relevant feature information, and matching the obtained relevant feature information with the music features in the music feature library to obtain the music features of the music file.

[0160] In some embodiments, the type of music feature is determined based on user selection or adjustment. That is, users can add or filter the music features they want. This allows for the generation of images that are more closely aligned with user preferences, further enhancing the user experience.

[0161] After generating the music features, in step 403, the image subject and visual template that match the music features are determined.

[0162] For example, identifying image themes that match the musical style and / or lyrics, and identifying visual templates that match at least one of the emotions, ideas, and rhythms expressed in the music. This allows for the determination of suitable image themes and visual templates from multiple dimensions of the music, further improving the matching between the generated images and the music.

[0163] In some embodiments, the image theme may include various styles of images, such as realism, retro, flat illustration, 3D illustration, psychedelic illustration, motion illustration, technological, industrial, cartoon, watercolor, sketch, oil painting, constructivism, children's picture book, traditional Chinese picture book, cyberpunk, pattern, ink painting, etc.

[0164] In some embodiments, the visual template may include various dynamic effects of the image, such as translation, rotation, scaling, fade-in / fade-out, unfold / fold, blink, bounce, transition, focus and defocus effects, etc.

[0165] In addition, after generating the music features, in step 404, the image features associated with the music features are determined.

[0166] In this embodiment of the invention, the execution order of steps 404 and 403 is not restricted. For example, steps 104 and 403 can be executed simultaneously or sequentially.

[0167] In some embodiments, image features may also be referred to as visual features or image visual features.

[0168] In some embodiments, image features associated with music features can be obtained in various ways. For example, image features whose relevance to music features meets preset conditions (e.g., relevance exceeds a preset threshold, i.e., strong correlation) can be extracted from an image database. In this way, extracting image features that meet preset conditions from an image database containing a large number of image features to generate images can improve the richness of the generated images and their matching with the music.

[0169] Alternatively, image features can be determined using artificial intelligence models. For example, text features corresponding to music features can be generated; these text features can then be input into an artificial intelligence model to obtain image features associated with the music features.

[0170] In this way, by using a model that derives image features from text features, it is possible to efficiently obtain image features associated with music features, thereby further improving the efficiency of image generation.

[0171] For example, the CLIP model can be used to output associated image features based on text with musical features. For details on the CLIP model, please refer to relevant technical resources; it will not be elaborated upon here.

[0172] In some embodiments, image features may include specific people, animals, things, environments, scenes, etc.

[0173] In some embodiments, when multiple musical features exist for a given piece of music, priorities can be set for these features. The image theme (e.g., image style), visual template (e.g., motion effects), and image features that match the highest-priority musical feature are then selected. Furthermore, the priority settings can be preset or set and adjusted by the user. Thus, when a generated music file has multiple musical features, the image theme, visual template, and image features that match the highest-priority musical feature are determined according to their priority, ensuring that the generated image matches the highest-priority musical feature.

[0174] After determining the image theme, visual template, and image features associated with the music features, in step 405, an image matching the music file is generated based on the image theme, visual template, and image features.

[0175] As mentioned above, since the determined image theme matches the music features, the visual template matches the music features, and the image features are associated with the music features, the generated image can match the music features in terms of visual attributes from different dimensions such as the overall image, the local image, and the image expression. Therefore, when the generated image is displayed while the music is playing, the user can intuitively feel the characteristics and connotation of the music from both visual and auditory perspectives, which can easily evoke resonance and effectively improve the user experience.

[0176] In some embodiments, the image can be generated based on artificial intelligence technology, for example, by generating an image that matches the music file based on a generative artificial intelligence model. This enables the efficient generation of images with high matching accuracy.

[0177] For example, images can be generated based on GAN models. Another example is the progressive generation of images using StyleGan and CLIP models. For details on GAN, StyleGan, and CLIP models, please refer to relevant technical resources; they will not be elaborated upon here.

[0178] In addition, graphics rendering techniques can be used when generating images.

[0179] The above provides a detailed explanation of the method for generating an image that matches the music file. That is, the method described in steps 401-405 above can be used, or other methods can be used to generate an image that matches the music file, and are not limited to the method described in steps 401-405 above.

[0180] In some embodiments, optionally, such as Figure 4 As shown, after generating an image that matches the music file, the method may further include:

[0181] Step 406: Play the music corresponding to the music file and display the generated image synchronously.

[0182] In this way, by displaying matching images while playing music, users can experience the music and highly matched images simultaneously through both auditory and visual senses, thus enhancing the user experience.

[0183] In some embodiments, the generated image that matches the music is not fixed, but can be dynamically updated as the music plays.

[0184] Figure 5 This is another block diagram of the multimedia processing method according to an embodiment of the present invention. For example... Figure 5 As shown, the method includes:

[0185] Step 501: Generate an image that matches the music file;

[0186] Step 502: During the playback of the music corresponding to the music file, the music features are dynamically updated in real time, and the image is dynamically updated according to the dynamically updated music features.

[0187] In step 501, generating an image that matches the music file can be done using the methods described in steps 401-405 above, or other methods can be used to generate an image that matches the music file, and are not limited to the methods described in steps 401-405 above.

[0188] In this way, dynamically updating images during music playback can present users with images that are dynamically related to the song's progress, climax, and the emotions it expresses. Users can simultaneously experience the changes in the song and images, further enhancing the user experience.

[0189] In some embodiments, the music features can be dynamically updated in real time based on at least one of the music's playback duration, structure, rhythm, and emotional changes. Based on these dynamically updated music features, at least one of the image's display parameters, display content, and display format can be dynamically updated. This allows for dynamic updates to image display parameters such as brightness and saturation, or the displayed content, or the display format (e.g., dynamic effects) based on the music's progression, climax, and emotional changes, further enhancing the display effect of the image changing synchronously with the music.

[0190] For example, if the lyrics of a song begin with a description of an "ancient castle," the generated image will revolve around the castle. As the music plays, if the lyrics mention a "quiet forest path," the image will automatically update to a winding path through the forest. This significantly enhances the user's intuitive understanding of how the image matches and changes synchronously with the music, further improving the user experience.

[0191] For example, the verses of the music have a relatively gentle rhythm, which is represented by a cat lying quietly to one side in the image. However, when the chorus of the music reaches its climax, the rhythm of the music becomes fast and exciting, and the sleeping cat in the image transforms into a cheetah running at full speed. This allows users to more intuitively and strongly feel the change in the rhythm of the music.

[0192] For example, when the music reaches its climax, the previously still figures in the image begin to dance, allowing users to more intuitively and strongly feel the arrival of the climax.

[0193] The following is an exemplary implementation of the multimedia processing method described above, including the following processing steps.

[0194] 1. Automatically analyzes song style and elements, including:

[0195] Audio decoding: Receives the audio signal stream of a song and decodes it to extract the audio signal that can be used for analysis;

[0196] Feature extraction: Using digital signal processing techniques to extract key features from audio signals, such as rhythm, pitch, harmony, and timbre;

[0197] Machine learning analysis: Utilizes trained machine learning models (such as neural networks) to analyze these features and determine the style and musical elements of a song; categorizes songs into predefined style categories, such as pop, rock, electronic, etc.

[0198] Music feature generation: Create music feature data containing style and musical elements, which will ultimately be used for subsequent image generation and user interface display.

[0199] 2. Real-time generation of images matching the song, including:

[0200] Music feature data reception: Receive music feature data from the module that automatically analyzes song style and music elements;

[0201] Content planning: Based on the music feature data, select one or more image themes that match the style and musical elements of the song, or match the song's music feature data;

[0202] Template selection: Choose a template from a variety of preset visual templates that matches the style and musical elements of the song, or choose a template that matches the characteristics of the music;

[0203] Image feature selection: Select image features from the image database that match the style and musical elements of the song, or select image features that match the musical features;

[0204] Image compositing: Using graphics rendering technology, images are generated based on visual templates, image themes, and image features. This step may involve color adjustment, shape deformation, texture application, etc.

[0205] Real-time adjustment: Dynamically adjust image details based on song duration and structure to reflect the song's highs and lows.

[0206] 3. Update images in real time based on the dynamic changes of the song, including:

[0207] Dynamic monitoring: Real-time monitoring of the song's audio output, capturing volume, rhythm, and other dynamic changes;

[0208] Sentiment Analysis: Using sentiment analysis algorithms, the emotional tendency of the current part of the song is inferred based on dynamic changes;

[0209] Image Adjustment: Based on the results of sentiment analysis, adjust image parameters such as color, brightness, and saturation to express the corresponding emotions;

[0210] Feedback loop: Continuously adjust the images to ensure that the emotions conveyed by the images and audio remain consistent throughout the song's playback.

[0211] 4. Supports multiple visual style options, including:

[0212] Style Definition: Define and develop various visual styles, including but not limited to abstract, modern, retro, and futuristic styles;

[0213] User settings: Allows users to select one or more preferred visual styles in the application's settings;

[0214] Style adaptation: Adjust the parameters in the image generation algorithm according to the style selected by the user to ensure that the generated image matches the user's style preference;

[0215] Dynamic application: While the song is playing, the images are dynamically updated according to the style selected by the user, providing a personalized visual experience.

[0216] In some embodiments, music tags corresponding to music features may also be displayed and updated during music playback.

[0217] Figure 6 This is another block diagram of the multimedia processing method according to an embodiment of the present invention. For example... Figure 6 As shown, the method includes:

[0218] Step 601: Obtain the music file;

[0219] Step 602: Generate the musical characteristics of the music file based on the music file;

[0220] Step 603: Display the music tags corresponding to the music characteristics; and

[0221] Step 604: During the playback of the music corresponding to the music file, the music features are dynamically updated in real time, and the music tags are dynamically updated according to the dynamically updated music features.

[0222] In this way, music tags are dynamically updated based on music playback, allowing users to intuitively perceive changes in music characteristics and further enhance the user experience.

[0223] In some embodiments, the specific implementation of steps 601 and 602 can refer to steps 401 and 402, or other methods can be used to obtain the music file and generate music features. Additionally, [the following can be added:] Figure 6 The method shown is the same as Figure 1 The methods and / or Figure 4 The methods shown can be used in combination, and the embodiments of the present invention do not limit this.

[0224] In some embodiments, music tags corresponding to musical features may include sound element tags, music style tags, and emotional tags expressed in the music.

[0225] For example, sound elements include the sounds of musical instruments, wind, rain, birdsong, and ocean waves.

[0226] For example, music genre tags include pop, rock, classical, jazz, electronic, folk, metal, and country.

[0227] For example, the emotional tags expressed in music include sweetness, romance, longing, pain, sadness, warmth, gratitude, remembrance, happiness, reluctance, excitement, loneliness, solitude, anger, and rebellion.

[0228] The following is an exemplary implementation of the multimedia processing method described above, including the following processing steps.

[0229] 1. Identify and label the main sound elements in the song, including:

[0230] Audio analysis: Receives the audio input of a song and uses spectrum analysis tools to extract information about frequency, duration, and timbre;

[0231] Feature extraction: Extracting key features from audio signals, such as harmonic structure, attack and attenuation characteristics, which help distinguish different sound elements (or musical elements);

[0232] Machine learning classification: Analyze the extracted features using a trained machine learning model (such as a convolutional neural network or an adversarial neural network) to identify the main sound elements in the song;

[0233] Tag generation: Based on the output of the machine learning model, create tags containing the names of the detected sound elements;

[0234] Tag Display: Displays generated sound element tags in the user interface and updates in real time to reflect changes in sound elements within the song.

[0235] 2. Identify and tag the style and mood labels of songs, including:

[0236] Audio processing: Perform audio analysis on the song to extract key features such as rhythm, pitch, and harmony;

[0237] Music analysis: Through music theory analysis, the tonality, chord progressions, and melodic lines of a song are determined, thereby inferring its style and emotion;

[0238] Style matching: Uses a preset music style database to match and identify the main style category to which the song belongs;

[0239] Sentiment Analysis: Utilizing emotion recognition algorithms to analyze the emotional tone of a song, such as happiness, sadness, excitement, or relaxation;

[0240] Tag display: Display identified style and mood tags in the user interface and ensure they accurately reflect the user's auditory experience.

[0241] 3. Real-time update of tag display, including:

[0242] Dynamic monitoring: Continuously monitors changes in the audio signal stream, capturing any updates to the song's structure and dynamics;

[0243] Tag adjustment: Adjust and update previously generated tags based on real-time analysis results of the audio signal stream;

[0244] User interface synchronization: Ensure that the label display in the user interface is synchronized with the current state of the song to provide timely and accurate information;

[0245] Feedback mechanism: If a new instrument or style change is detected, it is immediately fed back to the user interface to update the tag content.

[0246] 4. Provides label customization functionality, including:

[0247] User input: Allows users to input or select their favorite tags through an intuitive interface;

[0248] Personalized settings saving: Saves the user's custom tags in personal settings or configuration files;

[0249] Personalized applications: During song playback, a recommendation system and content display are set based on user-defined tags;

[0250] Interactive feedback: Provide a feedback mechanism to let users know how their custom tags affect the music playback experience.

[0251] 5. Recommend tags based on users' historical preferences, including:

[0252] Data analytics: Collecting and analyzing users' historical data, including song selections, tag preferences, and feedback;

[0253] Pattern recognition: Using data mining techniques to identify common patterns and preferences in user behavior;

[0254] Personalized recommendations: Based on identified patterns, new tags are recommended to users that match their historical preferences;

[0255] Continuous optimization: We continuously optimize the recommendation logic through machine learning algorithms to ensure that recommendations become more personalized and accurate over time.

[0256] In some embodiments, music tags can be adjusted to update music files, i.e., tuning functionality is achieved based on adjustments to music tags.

[0257] Figure 7 This is another block diagram of the multimedia processing method according to an embodiment of the present invention. For example... Figure 7 As shown, the method includes:

[0258] Step 701: Obtain the music file;

[0259] Step 702: Generate the musical characteristics of the music file based on the music file;

[0260] Step 703: Display the music tags corresponding to the music characteristics; and

[0261] Step 704: In response to the adjustment instruction for the music tag, adjust the music tag;

[0262] Step 705: Adjust the music parameters corresponding to the adjusted music tags;

[0263] Step 706: Update the music file according to the adjusted music parameters.

[0264] This enables music update functionality based on music tags, i.e., tuning functionality, thus improving the user experience.

[0265] In some embodiments, the specific implementation of steps 701 and 702 can refer to steps 401 and 402, or other methods can be used to obtain the music file and generate music features. Additionally, [the following can be added:] Figure 7 The method shown is the same as Figure 1 The method shown Figure 4 The method shown and Figure 6 At least one of the methods shown may be used in combination, and the embodiments of the present invention are not limited thereto.

[0266] In some embodiments, the specific implementation of steps 704-706 can refer to the specific implementation of steps 102-106, which will not be repeated here.

[0267] For example, similar to step 102, the adjustment instructions in step 704 are generated based on voice commands and / or operations on displayed controls. This allows for adjustments to images and / or music in multiple ways, making it convenient for the user.

[0268] For example, the equalizer (EQ) can be adjusted via voice, thereby regulating the music. Specifically, this can include the following processing steps:

[0269] User input parsing: Through speech recognition, the user's voice input adjustment commands are identified and converted into corresponding tag adjustment instructions;

[0270] EQ Search: In the established EQ library, EQ curves are categorized by labels. After the user inputs an adjustment command through speech recognition, the existing EQ curves that match the adjustment command are matched in the EQ library.

[0271] Matching EQ: Classifying and associating existing EQ curves with the smallest k-distance in the EQ library using a language model;

[0272] Applicable EQ: Receives user adjustment settings, applies the matching existing EQ curve to the audio signal, and generates the adjusted sound effect in real time;

[0273] Audio tuning interface call: Converts the matched existing EQ curve into instructions for the playback software;

[0274] Power amplifier processing: The power amplifier receives the instruction, the DSP processes it and reflects it on the audio signal stream, playing the adjusted sound effect in real time.

[0275] In some embodiments, after step 706, the method may further include: updating the image matching the updated music file according to the updated music file; playing the music corresponding to the updated music file and simultaneously displaying the updated image. This allows users to intuitively experience the effect of the adjustment from both auditory and visual perspectives. In some embodiments, user profiles can be pre-established, and images and / or music can be generated and / or adjusted based on these profiles. For example, a basic attribute profile can be generated based on demographic characteristics and economic and social attributes; a technology usage profile can be generated based on the user's smart device usage habits and AI application preferences; and a value and need profile can be generated based on the user's core motivations and pain point priorities.

[0276] In some embodiments, after adjusting the image and / or music, the user can save the adjusted image and / or music file and load the saved file when needed.

[0277] For example, a save button allows users to save the current image and / or music file, while a load button allows users to load the saved file.

[0278] In some embodiments, users may also provide a list or folder structure to organize and manage user settings or definitions.

[0279] In some embodiments, users can also customize the exported file, such as a JSON format file, to facilitate sharing and importing between different devices.

[0280] In some embodiments, users can share an image and / or music after adjusting it. For example, sharing the image or music with others via a web platform. Sharing the adjusted EQ.

[0281] In some embodiments, the user interface (UI) and user experience (UE) can also be optimized through the following measures:

[0282] Tag display: Tags need to be displayed in the user interface;

[0283] User-friendly tagging: Utilizing highly visual user interface designs with charts, sliders, and other adjustable adjustment methods;

[0284] User interface synchronization: Ensure that the label display is synchronized with the current state after the user adjusts the label. For example, when the user enters a custom label 1, label 1 needs to be displayed in the label list in a timely manner; when the user adjusts the weight of an existing label 2, the display effect of label 2 needs to be enhanced in the label list in a timely manner.

[0285] Layout optimization: Adjust the order of user preferences for tags in a timely manner to better align with user intuition and habits;

[0286] Usage tip: If the user's page is paused for more than 15 seconds, prompt the user about how the custom tag affects the music playback experience.

[0287] In some embodiments, pre-training can be used during model training to improve the model fit to achieve the requirement that the generated image matches the EQ being invoked now.

[0288] The multimedia processing method of this invention will be illustrated by specific examples below.

[0289] Examples of applicable scenarios:

[0290] Scenario 1: Commuting scenario. For example, a user turns on music while on a congested elevated highway, but truck horns and tire noise make the music sound muddy. This affects the listening experience, but the user lacks the ability to adjust the settings in real time, and traditional audio systems lack relevant adjustment ports.

[0291] Scenario 2: Self-driving scenario. For example, a user wants to immerse themselves in the scenery and atmosphere outside the window, creating a harmonious interplay between the interior and exterior. However, the user's verbal description is vague, and traditional voice recognition functions lack the necessary processing capabilities.

[0292] Scenario 3: Family Scenario. For example, when the whole family is traveling, they might want to adjust the sound settings with a simple description, such as "Adjust it so the elderly can listen more comfortably" or "Adjust the effect of the cartoon," but traditional sound effects functions lack relevant user-friendly adjustment options.

[0293] Scenario 4: Social Scenarios. For example, you might want to visualize music and share the generated images and EQ.

[0294] Example 1: A user operates the display screen of a terminal device (in-vehicle terminal) in the vehicle, selecting the music they want to play and / or edit; the in-vehicle terminal responds to the user's selection by retrieving the music file corresponding to the music locally, or by retrieving the music file corresponding to the music from a cloud server; the in-vehicle terminal generates music features for the obtained music file; the in-vehicle terminal, based on an artificial intelligence model, determines an image theme and visual template that match the music features; the in-vehicle terminal, based on the artificial intelligence model, extracts image features from an image database that meet preset conditions for relevance to the music features; the in-vehicle terminal generates an image that matches the music file based on the image theme, visual template, and image features.

[0295] The in-vehicle terminal plays the music and simultaneously displays the generated image. When the user wants to adjust the music, they can operate at least one image parameter control displayed on the music playback interface, or operate via voice, to adjust at least one image parameter. The in-vehicle terminal adjusts the music parameters corresponding to the adjusted image parameters and updates the music file accordingly. The in-vehicle terminal updates the image based on the adjusted image parameters. The in-vehicle terminal plays the music corresponding to the updated music file and simultaneously displays the updated image.

[0296] Example 2: A user operates the display screen of a terminal device (in-vehicle terminal) in the vehicle, selecting the music they want to play and / or edit; the in-vehicle terminal responds to the user's selection and sends the selected music information to a cloud server; the cloud server retrieves the corresponding music file from the music database; the cloud server generates music features for the obtained music file based on the music file; the cloud server determines the image theme and visual template matching the music features based on an artificial intelligence model; the cloud server extracts image features from the image database that meet preset conditions for relevance to the music features based on the artificial intelligence model; the cloud server generates an image matching the music file based on the image theme, visual template, and image features; the cloud server sends the music file and the matching image to the in-vehicle terminal.

[0297] The in-vehicle terminal plays the music and simultaneously displays a matching image. When the user wants to adjust the music, they can operate at least one image parameter control displayed on the music playback interface, or operate via voice. The in-vehicle terminal generates adjustment instructions for the image and music and sends them to the cloud server. The cloud server adjusts at least one image parameter and the corresponding music parameter according to the adjustment instructions. Furthermore, the cloud server updates the music file and the image according to the adjusted music parameters. The cloud server sends the updated music file and image to the in-vehicle terminal. The in-vehicle terminal plays the music corresponding to the updated music file and simultaneously displays the updated image.

[0298] Through the above embodiments, by utilizing the correlation between image parameters and music parameters, users can adjust the associated music parameters accordingly by adjusting the image parameters of the image that matches the music, thereby enabling music modification, i.e., realizing the tuning function based on image adjustment. Therefore, users can achieve the tuning function in an intuitive, simple, and novel way, enhancing the fun and ease of use, and improving the user experience.

[0299] Second aspect of the embodiments

[0300] A second aspect of the present invention provides a multimedia processing apparatus corresponding to the multimedia processing method described in the first aspect of the present invention; that is, the multimedia processing apparatus is a multimedia processing apparatus that implements the multimedia processing method of the first aspect embodiment. The functions of this multimedia processing apparatus can be referred to the description in the first aspect embodiment, and repeated content will not be specifically described again.

[0301] In some embodiments, the multimedia processing apparatus is a multimedia playback device or part of a multimedia playback device, or a mobile terminal or part of a mobile terminal, or an in-vehicle terminal or part of an in-vehicle terminal, or a cloud server or part of a cloud server. Alternatively, it may be a collection of the above-mentioned different devices, that is, different devices perform a portion of the steps respectively.

[0302] Figure 8 This is a block diagram of a multimedia processing device according to an embodiment of the present invention. Figure 8 As shown, the multimedia processing device 800 may include a processor 810 and a memory 820; the memory 820 is coupled to the processor 810. It is worth noting that this figure is exemplary; other types of structures may be used to supplement or replace this structure to implement telecommunications functions or other functions.

[0303] In one implementation, the processor 810 may be configured to: acquire an image matching a music file; adjust at least one image parameter of the image in response to an adjustment instruction for the image; adjust music parameters corresponding to the adjusted image parameters according to the adjusted image parameters; and update the music file according to the adjusted music parameters.

[0304] In this embodiment of the invention, the implementation of the functions of the processor 810 can refer to the description of the relevant steps in the embodiment of the first aspect of the invention, and will not be repeated here.

[0305] In addition, such as Figure 8 As shown, the multimedia processing device 800 may further include: a communication module 830, a display 840, and a speaker 850. It is worth noting that the multimedia processing device 800 does not necessarily include these components. Figure 8 All components shown; in addition, the multimedia processing device 800 may also include Figure 8 For components not shown, please refer to relevant technologies.

[0306] Through the above embodiments, by utilizing the correlation between image parameters and music parameters, users can adjust the parameters of the image that matches the music, and correspondingly adjust the associated music parameters, thereby modifying the music, that is, realizing the tuning function based on image adjustment; therefore, users can realize the tuning function in an intuitive, simple and novel way, enhancing the fun and ease of use, and improving the user experience.

[0307] Third aspect of the embodiments

[0308] A third aspect of the present invention provides a multimedia system that may include the multimedia processing apparatus described in the second aspect (i.e., the multimedia processing apparatus implementing the multimedia processing method of the first aspect). The content of the multimedia processing method can be referred to the description in the first aspect embodiment; repeated content will not be specifically described again.

[0309] In some embodiments, the multimedia system is a multimedia playback device or part of a multimedia playback device, or a mobile terminal or part of a mobile terminal, or an in-vehicle terminal or part of an in-vehicle terminal, or a cloud server or part of a cloud server.

[0310] Figure 9 This is a block diagram of a multimedia system according to an embodiment of the present invention. For example... Figure 9 As shown, the multimedia system 900 includes:

[0311] Multimedia processing device 901;

[0312] The sound output device 902 outputs music corresponding to the music file; and

[0313] Display device 903 displays an image matching the music file simultaneously when outputting music corresponding to the music file.

[0314] In some embodiments, the multimedia processing device 901 is the same as the multimedia processing device described in the second aspect embodiment;

[0315] In some embodiments, the sound output device 902 can be various types and various scenarios of sound output devices, such as speakers, headphones, etc., such as speakers in a vehicle;

[0316] In some embodiments, the display device 903 can be a display device of various types and in various scenarios, such as a monitor, such as the display screen of an in-vehicle terminal in a vehicle.

[0317] Through the above embodiments, by utilizing the correlation between image parameters and music parameters, users can adjust the parameters of the image that matches the music, and correspondingly adjust the associated music parameters, thereby modifying the music, that is, realizing the tuning function based on image adjustment; therefore, users can realize the tuning function in an intuitive, simple and novel way, enhancing the fun and ease of use, and improving the user experience.

[0318] Fourth aspect of the embodiment

[0319] This invention provides a vehicle that includes the multimedia processing device described in the second aspect of the embodiment or the multimedia system described in the third aspect of the embodiment.

[0320] In some embodiments, the vehicle may be a vehicle for various purposes, such as a passenger car, a truck, etc.

[0321] In some embodiments, the vehicle can be a vehicle with various power sources, such as automobiles, new energy vehicles, hybrid electric vehicles, etc.

[0322] When the multimedia processing method described in the first aspect of the present invention, the multimedia processing device described in the second aspect of the present invention, or the multimedia system described in the third aspect of the present invention are applied to vehicles, they can meet the needs of vehicle users for personalized, customized, and entertaining functions in the era of big data. Through natural language interaction, multimodal feedback, and emotional design, they achieve user-friendliness and technological accessibility, such as making tuning accessible to everyone, rather than just an "engineer's domain." Furthermore, they provide intuitive feedback and experience upgrades, making abstract adjustments "visible and tangible" through bidirectional enhancement of auditory and visual senses. In addition, they can also provide emotional value, enhancing the interactivity of vehicle functions to users through emotional connection, making the vehicle more "artificial intelligence-like" to users.

[0323] It should be noted that the limitations of each step involved in the embodiments of the present invention are not considered as limiting the order of steps without affecting the implementation of the specific solution. The steps listed first can be executed first, or they can be executed later, or they can even be executed simultaneously. As long as the solution can be implemented, they should be considered to fall within the protection scope of the present invention.

[0324] The present invention has been described above with reference to specific embodiments. However, those skilled in the art should understand that these descriptions are exemplary and not intended to limit the scope of protection of the present invention. Those skilled in the art can make various modifications and variations to the present invention based on its spirit and principles, and these modifications and variations are also within the scope of the present invention.

Claims

1. A multimedia processing method, characterized in that, The method includes: Get the image that matches the music file; In response to an adjustment instruction for the image, at least one image parameter of the image is adjusted; Based on the adjusted image parameters, the corresponding music parameters are adjusted; The music file is updated based on the adjusted music parameters.

2. The multimedia processing method according to claim 1, characterized in that, The method further includes: Based on the user's voice commands, generate adjustment instructions for the image, and / or, Based on the user's adjustments to the image parameters, the adjustment instructions for the image are generated.

3. The multimedia processing method according to claim 2, characterized in that, The voice commands are instructions for adjusting image parameters. And / or, the voice command expresses the user's requirements or expectations for the adjustment effect.

4. The multimedia processing method according to claim 3, characterized in that, The method also includes, The system receives the user's voice command regarding the desired adjustment effect, generates a recommended adjustment scheme for the image parameters of the image, and generates the adjustment command for the image after the user confirms or modifies it.

5. The multimedia processing method according to claim 1, characterized in that, The method further includes: A control that displays at least one of the image parameters; the user operates on the control of the displayed image parameters to generate the adjustment instructions for the image.

6. The multimedia processing method according to claim 5, characterized in that, The type and / or number of controls for the displayed image parameters can be set and / or adjusted by the user.

7. The multimedia processing method according to claim 1, characterized in that, Based on the adjusted image parameters, the music parameters associated with the image parameters are adjusted, including: The weights of the music parameters associated with the image parameters are adjusted based on the changes in the weights of the adjusted image parameters.

8. The multimedia processing method according to claim 1, characterized in that, The correspondence between the image parameters and the music parameters is preset. And / or, the correspondence between the image parameters and the music parameters can be set and / or adjusted by the user.

9. The multimedia processing method according to claim 1, characterized in that, The method further includes: Displays explanatory information regarding the relationship between the image parameters and the music parameters.

10. The multimedia processing method according to claim 1, characterized in that, The method further includes: The image is updated based on the adjusted image parameters; Play the music corresponding to the updated music file and simultaneously display the updated image.

11. The multimedia processing method according to any one of claims 1-10, characterized in that, The image parameters include at least one of the following: image brightness, contrast, saturation, image style, color, size, subject content, and dynamic effects, and / or... The music parameters include at least one of the following: pitch, timbre, tone, volume, music style, reverb, delay, gain, music stage, crossover sound pressure level, amplifier channel, and rhythm.

12. The multimedia processing method according to claim 1, characterized in that, Retrieve images that match the music file, including: Get music files; Based on the music file, generate the music features of the music file; Determine image themes and visual templates that match the music features; Determine the image features associated with the music features; An image matching the music file is generated based on the image theme, the visual template, and the image features.

13. The multimedia processing method according to claim 12, characterized in that, The method further includes: During the playback of the music corresponding to the music file, the music features are dynamically updated in real time, and the image is dynamically updated according to the dynamically updated music features.

14. The multimedia processing method according to claim 13, characterized in that, During the playback of the music corresponding to the music file, the music features are dynamically updated in real time. The image is dynamically updated based on the dynamically updated music features, including: The music features are dynamically updated in real time based on at least one of the music's playback duration, structure, rhythm, and emotional changes. Based on the dynamically updated music features, at least one of the image's display parameters, display content, and display format is dynamically updated.

15. The multimedia processing method according to claim 12, characterized in that, The method further includes: Display music tags corresponding to the aforementioned music features; and During the playback of the music corresponding to the music file, the music features are dynamically updated in real time, and the music tags are dynamically updated based on the dynamically updated music features.

16. The multimedia processing method according to claim 15, characterized in that, The method further includes: In response to the adjustment instruction for the music tag, the music tag is adjusted; Based on the adjusted music tags, the music features corresponding to the music tags are adjusted; The music file is updated based on the adjusted music characteristics.

17. The multimedia processing method according to claim 1, 12, or 16, characterized in that, The method further includes: Based on the analysis of collected user data, recommended adjustment schemes related to images and / or music files are generated and displayed.

18. A multimedia system, characterized in that, The multimedia system has a multimedia processing apparatus for implementing any of the multimedia processing methods according to claims 1-17, and the multimedia system further comprises: Audio output devices, which output music files corresponding to the audio; and The display device simultaneously displays an image matching the music file while outputting the music file.

19. A vehicle, characterized in that, The vehicle includes the multimedia system of claim 18.

Images