A method for synthesizing audio-visual content synchronized sound effects

Abstract

The present invention discloses a method for synthesizing synchronized sound effects for audio-visual content, and relates to the technical field of audio-visual content processing. The method for synthesizing synchronized sound effects for audio-visual content extracts features from each frame of image data to be synthesized, identifies key frames for generating sound effects in combination with a pre-trained visual event recognition model, extracts triggering event information, and further analyzes the spatial sound image positioning index and emotional sound effect regulation index of the corresponding frame image. On this basis, the sound effect matching index of each frame image is calculated, and interval matching is performed with a preset sound effect segment to achieve accurate selection and synthesis of sound effects. The present invention introduces an image feature set and, with the help of a pre-trained visual event recognition model, constructs a complete recognition path for image event dynamics. It can not only accurately identify event frames with sound effect requirements in the absence of significant scene switching in visual information, but also output structured information such as event area image data, event type, and trigger time.

Description

Technical Field

[0001] The present invention relates to the technical field of audio-visual content processing, and in particular to a method for synthesizing synchronized sound effects of audio-visual content. Background Art

[0002] Against the backdrop of rapid developments in multimedia content generation and processing technology, improving the perceptual consistency between images and audio has become an important research direction in the field of audiovisual content enhancement. Currently, many video editing or intelligent synthesis systems rely on manual annotation of events and manual matching of sound effect clips, which presents problems such as low efficiency, poor adaptability, and strong subjectivity. Especially in scenes involving sudden, emotional, or spatially dynamic images, existing sound effect generation methods often have difficulty accurately reflecting the spatial position and emotional intensity of image events, resulting in distorted audiovisual content expression and affecting the user's immersive experience. With the integrated application of technologies such as image recognition, audio synthesis, and multimodal learning, building a technical path that can automatically identify key events in images and synchronously synthesize matching sound effects has become a key breakthrough in improving the expressiveness of video content and the emotional rendering effect.

[0003] The limitations of existing technologies include at least the following problems: most sound effect synthesis methods rely mainly on fixed timestamps, preset rules or global image change trends to trigger sound effect generation, but ignore the actual occurrence time and regional differences of local events in the image. This method based on coarse time drive or full-frame processing is difficult to capture key frames in the image sequence that actually have sound effect requirements. In audiovisual content, local events with sound effect significance are often instantaneous triggers such as object collisions, character turns, and action starts. The spatial positions of these events and image emotional changes are often highly local and non-continuous. If only low-resolution indicators such as the average brightness of the entire frame and the global motion amplitude are used to determine whether sound effects need to be generated, a large number of non-key frames will be misjudged as sound effect frames, or key frames will be omitted, which will directly cause the sound effects to be mistimed or missing. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a method for synthesizing synchronized sound effects for audiovisual content, which solves the problem that the existing technology lacks the ability to accurately identify the frames where image events occur, thereby leading to inaccurate sound effect matching in audiovisual content and a fragmented experience.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for synthesizing synchronized sound effects of audio-visual content, comprising the following steps: obtaining several frames of image data to be synthesized, and performing feature extraction on each frame to be synthesized to obtain an image feature set for each frame of image to be synthesized; inputting the image feature set of each frame of image to be synthesized into a pre-trained visual event recognition model, identifying several frames of images for which sound effects need to be generated, and outputting corresponding trigger event frame information; based on the image feature set and the trigger event frame information, analyzing the spatial sound image positioning index and emotional sound effect regulation index of each frame of image for which sound effects need to be generated; based on the spatial sound image positioning index and the emotional sound effect regulation index, matching the sound effect clips of each frame of image for which sound effects need to be generated, and performing sound effect synthesis processing.

[0006] Furthermore, each frame of image data includes pixel values ​​and two-dimensional coordinates of several pixel points, and the image feature set includes image brightness mean, color mutation rate and inter-frame optical flow vector modulus.

[0007] Furthermore, the specific steps for obtaining the image feature set of each frame image to be synthesized are as follows: performing average grayscale processing on each frame image to be synthesized respectively to obtain the image brightness mean of each frame image to be synthesized; performing color space histogram analysis processing on adjacent frame images to be synthesized to obtain the color mutation rate of each frame image to be synthesized; performing optical flow estimation averaging processing on adjacent frame images to be synthesized to obtain the inter-frame optical flow vector modulus value of each frame image to be synthesized.

[0008] Furthermore, the trigger event frame information includes trigger time, event type and event area image data, the event area image data includes pixel values ​​and two-dimensional coordinates of several event pixel points, and the visual event recognition model includes a frame-level feature encoding layer, a time series modeling layer, an attention focus layer, and an event classification and positioning output layer.

[0009] Furthermore, the specific steps of identifying several frames of images for which sound effects need to be generated and outputting the corresponding trigger event frame information are as follows: in the frame-level feature coding layer of the visual event recognition model, the image feature set of each frame of image to be synthesized is received, and field-by-field encoding is performed to output the image coding feature vector of the corresponding frame; in the time series modeling layer of the visual event recognition model, the image coding feature vectors of each frame of image are arranged in chronological order, a time series feature tensor is constructed, and the time correlation feature vector of each frame of image is extracted; in the attention focusing layer of the visual event recognition model, the time correlation feature vector of each frame of image is read, and the difference score of the feature change rate between it and the adjacent frames is performed, and the frames with the difference score higher than the set threshold are extracted as candidate trigger event frames; in the event classification and positioning output layer of the visual event recognition model, according to the image coding feature vectors of the candidate trigger event frames, the corresponding trigger event frame information of several frames of image for which sound effects need to be generated is analyzed and output.

[0010] Furthermore, the specific steps of analyzing the spatial sound image localization index of each frame image for which sound effects need to be generated are as follows: based on the event area image data, analyzing the edge disturbance intensity and the coordinates of the center point of the event area of ​​each frame image for which sound effects need to be generated; based on the two-dimensional coordinates of several pixel points of each frame image for which sound effects need to be generated, analyzing the image center coordinates of the corresponding frame image, and analyzing the center coordinate offset value in combination with the coordinates of the center point of the event area of ​​the corresponding frame image; based on the event area image data, analyzing the event area ratio value of each frame image for which sound effects need to be generated; reading the inter-frame optical flow vector modulus value of each frame image for which sound effects need to be generated, and performing a comprehensive analysis in combination with the edge disturbance intensity, center coordinate offset value and event area ratio value of the corresponding frame image to obtain the spatial sound image localization index of each frame image for which sound effects need to be generated.

[0011] Furthermore, the specific formula for calculating the spatial sound image localization index of a frame of image for which sound effects need to be generated is as follows:

[0012] ;

[0013] in, is the spatial sound image localization index of a frame of image for which sound effects need to be generated, is the edge disturbance intensity of a frame image for which sound effects need to be generated, is the edge disturbance response coefficient stored in the database, is the modulus of the optical flow vector between frames of a certain frame image for which sound effects need to be generated, is the optical flow response coefficient stored in the database, is the center coordinate offset value of a frame image for which sound effects need to be generated. is the center shift response coefficient stored in the database, is the area ratio of the event region of a frame image where sound effects need to be generated. is the area ratio response coefficient stored in the database.

[0014] Furthermore, the specific steps for analyzing the emotional sound effect regulation index of each frame image for which sound effects need to be generated are as follows: based on the event area image data, the local texture disturbance ratio and the maximum color gradient value of each frame image for which sound effects need to be generated are analyzed; the image brightness mean and the color mutation rate of each frame image for which sound effects need to be generated are read, and a comprehensive analysis is performed in combination with the texture disturbance ratio and the maximum color gradient value of the corresponding frame image to obtain the emotional sound effect regulation index of each frame image for which sound effects need to be generated.

[0015] Furthermore, the specific formula for calculating the emotional sound effect adjustment index of a frame of image for which sound effects need to be generated is as follows:

[0016] ;

[0017] in, is the emotional sound effect adjustment index of a frame image for which sound effects need to be generated, is the color mutation rate of a frame image for which sound effects need to be generated, is the color mutation response coefficient stored in the database, is the texture perturbation ratio of a frame image for which sound effects need to be generated, is the texture disturbance response coefficient stored in the database, is the interaction response coefficient stored in the database, is the maximum color gradient value of a frame image for which sound effects need to be generated. is the color gradient response coefficient stored in the database, is the average brightness value of a frame of image for which sound effects need to be generated, is the image brightness response coefficient stored in the database.

[0018] Furthermore, based on the spatial sound image positioning index and the emotional sound effect adjustment index, the specific steps for matching the sound effect segment of each frame image for which sound effects need to be generated are as follows: read the spatial sound image positioning index and the emotional sound effect adjustment index of each frame image for which sound effects need to be generated, and perform weighted analysis to obtain the sound effect matching index of each frame image for which sound effects need to be generated; judge and analyze the sound effect matching index of each frame image for which sound effects need to be generated with several preset sound effect matching intervals, and each sound effect matching interval corresponds to a sound effect segment; and use the sound effect segment corresponding to the sound effect matching index in a preset sound effect matching interval as the generated sound effect of the image.

[0019] The present invention has the following beneficial effects:

[0020] (1) This audio-visual content synchronization sound effect synthesis method introduces an image feature set consisting of image brightness mean, color mutation rate and inter-frame optical flow vector modulus, and with the help of a pre-trained visual event recognition model, constructs a complete recognition path for image event dynamics from frame-level feature encoding, time series modeling, attention focus to event classification and positioning output. It is significantly better than the existing passive triggering method based only on frame difference or threshold judgment. This method can not only accurately identify event frames with sound effect requirements when there is no significant scene switching in visual information, but also support the subsequent refined modeling of audio-visual mapping and emotion regulation calculation by outputting structured information such as event area image data, event type and trigger time, effectively avoiding the problem of sound effect triggering delay, dislocation or omission, improving the accuracy of sound response of local events in multi-frame images, and enhancing the synchronization expressiveness and user immersion of the final audio-visual content.

[0021] (2) This audiovisual content synchronization sound effect synthesis method constructs a spatial sound image positioning index and an emotional sound effect adjustment index. The former is modeled by integrating edge disturbance intensity, center of mass offset, regional area ratio and optical flow vector modulus value, and the latter is modeled by integrating local texture disturbance ratio, maximum color gradient, image brightness and color mutation features. It significantly improves the spatial matching and emotional perception capabilities of sound effect clips. By introducing response coefficients such as edge disturbance response coefficient and texture disturbance response coefficient, this method avoids the limitations of traditional static mapping sound effect synthesis in regional adaptation and emotional differentiation, and realizes the fine-grained expression of local dynamic features and their actual quantification in sound effect design, so that the sound effect content can not only match the visual position, but also adapt to the emotional tension brought by picture changes, effectively supporting multi-dimensional sound and picture linkage under complex content.

[0022] (3) The audio-visual content synchronization sound effect synthesis method extracts the spatial sound image positioning index and the emotional sound effect adjustment index, sets a weighted fusion to form a sound effect matching index, and performs mapping judgment with multiple preset sound effect matching intervals. Each matching interval is bound to a specific sound effect segment. While ensuring the rationality of the judgment, this mechanism realizes the dynamic allocation of sound effect segments and the optimal scheduling of resources, avoiding the problem of segment duplication and synthesis imbalance caused by fixed rule matching. During the sound effect synthesis process, the selected segment will be inserted into the target audio track according to its corresponding trigger event time, and maintain timing synchronization and audio envelope continuity, ultimately realizing the full process linkage from event recognition to audio generation, improving the expression richness and rhythm control ability of audio-visual content in complex scenes, and significantly enhancing the overall scalability and practicality of the system.

[0023] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flow chart of a method for synthesizing audio-visual content synchronized sound effects.

[0025] Figure 2 The present invention is a flowchart of the specific steps of obtaining an image feature set for each frame of an image to be synthesized in a method for synthesizing synchronized sound effects of audiovisual content.

[0026] Figure 3 The present invention is a flowchart of the specific steps of outputting corresponding trigger event frame information in a method for synthesizing audio-visual content synchronized sound effects. DETAILED DESCRIPTION

[0027] See also Figure 1 The embodiment of the present invention provides a method for synthesizing audio-visual content synchronized sound effects, comprising the following steps:

[0028] Acquire several frames of image data to be synthesized, and perform feature extraction on each frame to obtain an image feature set for each frame of image to be synthesized; input the image feature set of each frame of image to be synthesized into a pre-trained visual event recognition model, identify several frames of image for which sound effects need to be generated, and output corresponding trigger event frame information; based on the image feature set and the trigger event frame information, analyze the spatial sound image positioning index and emotional sound effect regulation index of each frame of image for which sound effects need to be generated; based on the spatial sound image positioning index and the emotional sound effect regulation index, match the sound effect clip of each frame of image for which sound effects need to be generated, and perform sound effect synthesis processing.

[0029] Among them, the specific steps of performing sound effect synthesis processing are as follows: according to the trigger event frame information of each frame image, its trigger time field is extracted, and according to the matched sound effect segment, the starting playback position of the sound effect segment is time-aligned with the trigger time of the corresponding image frame; then, the audio resource path of the sound effect segment is called in the synthesized sound track, and according to the spatial sound image positioning index of the current frame image, the spatial sound image offset parameters of the sound effect segment in the sound track are set to adjust the spatial display position of the sound effect in the left and right channels; at the same time, according to the emotional sound effect adjustment index of the frame image, the loudness control value and rhythm fluctuation curve parameters of the sound effect segment are set to adjust the playback intensity and rhythm changes of the segment in the synthesized output; after completing the above parameter configuration, the sound effect segment is inserted into the specified position of the target sound track, and the mixing synthesis is performed according to the set sound effect hierarchy structure to generate a sound track segment containing the sound effect content corresponding to the target frame image.

[0030] Specifically, each frame of image data includes pixel values ​​and two-dimensional coordinates of several pixel points, and the image feature set includes image brightness mean, color mutation rate and inter-frame optical flow vector modulus.

[0031] like Figure 2 As shown, the specific steps for obtaining the image feature set of each frame image to be synthesized are as follows: perform average grayscale processing on each frame image to be synthesized respectively to obtain the image brightness mean of each frame image to be synthesized, which is specifically: first, read the pixel values ​​of all pixels of each frame image to be synthesized and convert them into grayscale values. The grayscale value of each pixel is calculated by the standard weighting method. The commonly used weighting factors are: the red channel multiplied by 0.2989, the green channel multiplied by 0.5870, and the blue channel multiplied by 0.1140. Then, calculate the average of the grayscale values ​​of all pixels in the current frame image to obtain the image brightness mean of the frame image. This brightness mean reflects the overall brightness of the image.

[0032] The color space histogram analysis is performed on the adjacent frame images to be synthesized to obtain the color mutation rate of each frame image to be synthesized. Specifically, the current frame image to be synthesized and its adjacent frame image are obtained. By analyzing the color space histograms of these two frames, the degree of difference in color distribution of each frame image is calculated. The difference can be measured by methods such as chi-square distance or histogram intersection. In this way, the color mutation rate of each frame image is obtained, which reflects the magnitude of the color change of the image.

[0033] The optical flow estimation and averaging processing is performed on the adjacent frame images to be synthesized to obtain the inter-frame optical flow vector modulus of each frame image to be synthesized. Specifically, the current frame image and its previous frame image are obtained, and the optical flow method is used to estimate the displacement (motion vector) of each pixel between the two frames. The optical flow algorithm calculates the motion vector (u, v) of each pixel through the movement of the pixel in space. Then, the modulus of the optical flow vector of each pixel is calculated to represent the motion intensity of the pixel. Finally, the optical flow modulus values ​​of all pixels are averaged to obtain the inter-frame optical flow vector modulus value of the frame image, which reflects the motion intensity and change speed of the image.

[0034] In this implementation, the image feature set is formed by extracting the image brightness mean, color mutation rate and inter-frame optical flow vector modulus respectively, which significantly improves the comprehensive perception ability of the dynamic change characteristics of the image. The image brightness mean reflects the overall visual brightness and darkness level, which helps to identify the lighting characteristics of the scene changes; the color mutation rate evaluates the color distribution changes through histogram differences, and accurately captures sudden or significant visual events in the picture; the inter-frame optical flow vector modulus quantifies the motion intensity at the pixel level in the image, and effectively perceives the dynamic behavior of objects in the scene. The three work together to build a stable, sensitive and directional image feature foundation, enhancing the accuracy and response timeliness of subsequent event recognition and sound effect generation.

[0035] Specifically, the trigger event frame information includes the trigger time, event type and event area image data. The event area image data includes the pixel values ​​and two-dimensional coordinates of several event pixel points. The visual event recognition model includes a frame-level feature encoding layer, a time series modeling layer, an attention focus layer, and an event classification and positioning output layer.

[0036] The pre-training steps of the visual event recognition model are as follows:

[0037] First, an image sequence event dataset is constructed. The dataset consists of multiple short-term image sequence samples, each sample consists of continuously shot image frames, each frame of the image contains complete pixel values ​​and two-dimensional coordinate fields, and is equipped with a trigger event field annotated frame by frame. Specifically, each frame of the image is manually or semi-automatically annotated to determine whether it contains a trigger event for generating sound effects. If it does, the event type label of the frame image (such as "falling", "broken", "explosion", etc.) and the event area image data, that is, the pixel values ​​and two-dimensional coordinates of all pixels in the corresponding area. Each image sequence sample in the training dataset is also equipped with a real time axis for accurate calibration of the event trigger time to ensure that the time field in the subsequent model output is traceable and verifiable.

[0038] Before training, all structural layers of the visual event recognition model need to be initialized. The input of the frame-level feature coding layer is three fields: image brightness mean, color mutation rate, and inter-frame optical flow vector modulus. A field-level coding network is used to construct a linear mapping layer and a nonlinear activation unit for each field, outputting a vector embedding of uniform length. The initialization parameters use the Xavier initialization method, and a small batch of image samples is used to pre-train this layer separately. The training goal is to minimize the reconstruction error of each field in the same frame to enhance the expression independence and discrimination ability between fields. In this stage, only the frame-level feature coding layer is trained, and its output serves as the basic input for the subsequent modeling layer.

[0039] After completing the pre-training of the feature encoding layer, the training process is extended to the time series modeling layer, the attention focusing layer, and the event classification and localization output layer. An end-to-end joint training method is adopted. First, the bidirectional gated recurrent unit (Bi-GRU) is used to construct the time modeling layer to perform temporal correlation modeling on the encoding features of the image frame sequence; then, the attention focusing layer performs local difference analysis on the temporal features to extract potential trigger nodes; finally, the event classification and localization output layer performs event type classification and region localization processing on the trigger frame. The training objectives include two items: one is to minimize the event classification loss function (using cross entropy loss), and the other is to minimize the event region localization error (using a weighted combination of IoU loss and bounding box offset loss). During the training process, the entire model gradually converges to a state that can accurately output trigger event frame information based on the Adam optimizer.

[0040] like Figure 3As shown, the specific steps of identifying several frames of images for which sound effects need to be generated and outputting the corresponding trigger event frame information are as follows: in the frame-level feature coding layer of the visual event recognition model, the image feature set of each frame of image to be synthesized is received, and field-by-field encoding is performed to output the image coding feature vector of the corresponding frame, which is specifically: the image feature set of each frame of image to be synthesized is received, and the three fields of image brightness mean, color mutation rate and inter-frame optical flow vector modulus are independently encoded in turn. The encoding method is a field-level transformation network constructed based on a fully connected mapping structure, which maps the original value of each field to an embedded vector of a unified dimension and then splices and fuses them to form an image coding feature vector of the current frame image, which is used to represent the multi-dimensional expression characteristics of the frame at the static attribute level.

[0041] In the time series modeling layer of the visual event recognition model, the image encoding feature vectors of each frame are arranged in chronological order to construct a time series feature tensor, and the time correlation feature vector of each frame is extracted. Specifically, the image encoding feature vectors of each frame output by the frame-level feature encoding layer are arranged in frame order to construct a time series feature tensor; then a sequence modeling network based on a bidirectional gated recurrent unit (Bi-GRU) is used to perform bidirectional propagation processing on the feature tensor, extract the dynamic correlation features between each frame and its temporal neighboring frames, and output the time correlation feature vector of each frame to represent the dynamic state of the frame during the event evolution process.

[0042] In the attention focus layer of the visual event recognition model, the temporal correlation feature vector of each frame is read, and the difference score of the feature change rate between it and the adjacent frames is performed, and the frames with the difference score higher than the set threshold are extracted as candidate trigger event frames. Specifically, the temporal correlation feature vector of each frame is input, and the feature change rate between it and the previous and next frames is scored using a scoring model based on the weighted attention mechanism; the scoring model calculates the modulus of the temporal correlation feature difference vector between the current frame and the adjacent frames, and combines the attention weight mapping mechanism to assign higher scores to frames with significant difference changes; finally, the frame image with the difference score higher than the set threshold is extracted as the candidate trigger event frame, indicating that the frame may correspond to the key node of the event mutation in the image.

[0043] In the event classification and positioning output layer of the visual event recognition model, the corresponding trigger event frame information of several frames of images for generating sound effects is analyzed and output according to the image coding feature vector of the candidate trigger event frame. Specifically, the image coding feature vector of the candidate trigger event frame is received and input into the multi-classification event recognition network and the region positioning network for processing respectively; wherein, the event recognition network performs feature discrimination on the image coding feature vector based on the multi-layer perceptron structure, and outputs the event type category label to which the frame image belongs; the region positioning network extracts the event region image data in the frame based on the pixel-level attention mapping and image back projection mechanism of the candidate frame image, including the pixel value of the event pixel point and its two-dimensional coordinates; finally, the trigger event frame information including the trigger time, event type and event region image data is output for subsequent sound effect matching and synthesis processing.

[0044] In this implementation, by constructing a multi-level visual event recognition model, accurate recognition of key image frames for generating sound effects and extraction of event areas are achieved. The model pre-training process is meticulously layered. First, static and dynamic features such as image brightness, color mutation rate and inter-frame optical flow are encoded, and then temporal modeling is performed through Bi-GRU to capture the temporal features in event evolution. Subsequently, the attention mechanism is used to accurately focus on frames with significant feature changes to enhance the response sensitivity to sudden events. Finally, the classification and positioning output layer can not only identify the event type, but also extract pixel data of the event area, providing reliable semantic support and spatial anchors for subsequent sound effect generation. This processing flow significantly improves the accuracy of event recognition and the accuracy of time positioning, avoids the sound effect mismatch problem caused by blurred event boundaries and inaccurate frame-level judgment in traditional methods, and provides a highly timely and robust technical foundation for audio-visual content synthesis.

[0045] Specifically, the specific steps for analyzing the spatial sound image localization index of each frame image for which sound effects need to be generated are as follows: based on the event area image data, analyzing the edge disturbance intensity and the coordinates of the centroid of the event area of ​​each frame image for which sound effects need to be generated; based on the two-dimensional coordinates of several pixel points of each frame image for which sound effects need to be generated, analyzing the image center coordinates of the corresponding frame image (i.e., calculating the arithmetic mean of all x-direction and y-direction coordinate values), and analyzing the center coordinate offset value in combination with the coordinates of the centroid of the event area of ​​the corresponding frame image; based on the event area image data, analyzing the event area ratio value of each frame image for which sound effects need to be generated (i.e., the ratio of the number of pixels in the event area to the number of pixels in the image); reading the inter-frame optical flow vector modulus value of each frame image for which sound effects need to be generated, and performing a comprehensive analysis in combination with the edge disturbance intensity, the center coordinate offset value, and the event area ratio value of the corresponding frame image to obtain the spatial sound image localization index of each frame image for which sound effects need to be generated.

[0046] The specific steps for analyzing the edge perturbation intensity of each frame image for which sound effects need to be generated are as follows: reading the image data of the identified event area in the current frame image, extracting the pixel values ​​and corresponding two-dimensional coordinate values ​​of all pixels in the area; then, using an edge detection algorithm (such as the Sobel operator) to perform an edge extraction operation on the area, obtaining the gradient direction and intensity distribution of the edge pixels, and calculating the standard deviation of the change rate of all edge pixels in the gradient direction as the edge perturbation intensity of the frame image.

[0047] The specific steps for analyzing the coordinates of the centroid of the event area of ​​each frame image for which sound effects need to be generated are as follows: based on the two-dimensional coordinates of all pixel points in the event area, the arithmetic mean of all x-direction and y-direction coordinate values ​​is calculated to obtain the coordinates of the centroid of the event area as the structural feature representing the spatial center position of the area.

[0048] The specific formula for calculating the spatial sound image localization index of a frame of image for which sound effects need to be generated is as follows:

[0049] ;

[0050] in, is the spatial sound image localization index of a frame of image for which sound effects need to be generated, is the edge disturbance intensity of a frame image for which sound effects need to be generated, is the edge disturbance response coefficient stored in the database, is the modulus of the optical flow vector between frames of a certain frame image for which sound effects need to be generated, is the optical flow response coefficient stored in the database, is the center coordinate offset value of a frame image for which sound effects need to be generated. is the center shift response coefficient stored in the database, is the area ratio of the event region of a frame image where sound effects need to be generated. is the area ratio response coefficient stored in the database.

[0051] It should be explained that the edge disturbance response coefficients stored in the database , optical flow response coefficient , center shift response coefficient , area ratio response coefficient The specific steps to obtain are as follows:

[0052] For the edge perturbation response coefficient: select the image frames in the sample set that are marked as those for which sound effects need to be generated, extract the image data of the event area in each sample image in turn, call the edge detection operator (such as Sobel) to calculate the edge gradient values ​​of all pixels in the event area, and calculate the average and standard deviation of the gradient direction change rate of all edge pixels in the area. The calculation result is the edge perturbation intensity value of the current sample. Subsequently, based on the fitting relationship between the edge perturbation intensity of all samples and their actual sound and image offset, the least squares method is used for regression analysis, and the linear weight term of the edge perturbation intensity in the regression equation is extracted as the edge perturbation response coefficient. The data type is a floating point number with a precision of three decimal places.

[0053] The method for obtaining the optical flow response coefficient is as follows: perform frame-by-frame optical flow estimation processing on each group of training images containing continuous frames, obtain the motion vector values ​​of all pixels in the entire frame image, and calculate the full-frame average value of the optical flow modulus value as the inter-frame motion intensity index of the frame. Then, the motion intensity is matched one-to-one with the amplitude of the sound and image offset direction of the final generated sound effect, fit their change relationship and extract the fitting coefficient of the optical flow intensity term as the optical flow response coefficient. The data type is fixed-point real number, and four significant digits are retained.

[0054] The center offset response coefficient is obtained as follows: read all pixel coordinate values ​​of the event area in each frame of the image, calculate the coordinates of the center point of the area, and calculate the Euclidean distance with the geometric center coordinates of the entire frame of the image. After normalization, it is used as the center offset value. Then, a linear fit is performed on the center offset values ​​of all training samples and their sound image localization error amplitude. The offset slope in the fitting result is the center offset response coefficient, which is expressed as the percentage of sound image offset corresponding to each pixel offset, and the accuracy is retained to the 0.001 level.

[0055] The area ratio response coefficient is obtained as follows: the number of valid pixels in the event area in the training sample is counted and the ratio is calculated with the total number of pixels in the frame image to obtain the area ratio of the event area. The area ratio value is then associated with the loudness gain parameter of the final synthesized sound effect clip for modeling. A logarithmic function is used to fit its trend, and the slope of the logarithmic curve of the area ratio factor is extracted as the area ratio response coefficient.

[0056] The specific implementation example of calculating the spatial sound image localization index of a frame of image for which sound effects need to be generated is as follows, including the following parameters:

[0057] The edge perturbation intensity of a frame of image for which sound effects need to be generated is approximately: 1.215.

[0058] The edge disturbance response coefficient stored in the database is approximately: 0.395.

[0059] The modulus of the inter-frame optical flow vector of a frame image for which sound effects need to be generated is approximately 0.982.

[0060] The optical flow response coefficient stored in the database is approximately: 0.336.

[0061] The center coordinate offset value of a frame image for which sound effects need to be generated is approximately: 0.654.

[0062] The center shift response coefficient stored in the database is approximately: 0.431.

[0063] The event area ratio of a frame image for which sound effects need to be generated is approximately 0.148.

[0064] The area ratio response coefficient stored in the database is approximately: 0.572.

[0065] Substituting the above data into the specific formula for calculating the spatial sound image localization index of a frame of image for which sound effects need to be generated, we obtain:

[0066] The spatial sound image localization index of a frame image for which sound effects need to be generated = (((1.215)^0.395) / (1+exp(-0.336×0.982)))×ln(1+0.431×((0.654)^)2)×(0.572×((0.148)^(1 / 2)))≈0.023391.

[0067] In this implementation, by constructing a spatial sound and image localization index, the system integrates multiple physically measurable image structure parameters such as edge disturbance intensity, inter-frame optical flow modulus, center coordinate offset value and event area ratio, effectively enhancing the mapping logic between image events and spatial sound effects. The index not only takes into account the edge complexity and dynamic change intensity of the event area in the image, but also combines the spatial distribution offset and coverage ratio of the event area in the image frame, comprehensively reflecting the spatial significance and sound and image triggering characteristics of the event. The weight coefficients of each parameter are obtained through mathematical fitting based on a large number of labeled samples, with a clear data source and response mechanism, ensuring the stability and traceability of the index calculation. Through this structured modeling method, the sound and image direction and sound field focus position that should be matched for each frame of the image can be accurately predicted, significantly improving the spatial realism and immersion of the sound effect synthesis, and avoiding the sound and image drift or positioning error caused by the lack of event space information in traditional solutions.

[0068] Specifically, the specific steps for analyzing the emotional sound effect regulation index of each frame image for which sound effects need to be generated are as follows: based on the event area image data, the local texture disturbance ratio and the maximum color gradient value of each frame image for which sound effects need to be generated are analyzed; the image brightness mean and the color mutation rate of each frame image for which sound effects need to be generated are read, and a comprehensive analysis is performed in combination with the texture disturbance ratio and the maximum color gradient value of the corresponding frame image to obtain the emotional sound effect regulation index of each frame image for which sound effects need to be generated.

[0069] Among them, the specific steps of analyzing the local texture disturbance ratio of each frame image where sound effects need to be generated are: reading the image data of the event area where sound effects need to be generated in the current frame image, and extracting the pixel values ​​and corresponding two-dimensional coordinate values ​​of all pixels in the area; for obtaining the local texture disturbance ratio, a sliding window method is used to perform gray-level co-occurrence matrix (GLCM) calculation on the event area, and three texture indicators of contrast, entropy and energy are extracted in each local window. The proportion of windows in which the fluctuation of each indicator exceeds the set threshold in the entire event area is counted as the local texture disturbance ratio of the area.

[0070] The specific steps for analyzing the maximum color gradient value of each frame image for generating sound effects are as follows: convert the event area image to the HSV color space, then calculate the gradient values ​​of the hue (H) channel in the horizontal and vertical directions respectively, and select the maximum value of all hue gradient values ​​in the area as the maximum color gradient value of the current frame image.

[0071] The specific formula for calculating the emotional sound effect adjustment index of a frame of image for which sound effects need to be generated is as follows:

[0072] ;

[0073] in, is the emotional sound effect adjustment index of a frame image for which sound effects need to be generated, is the color mutation rate of a frame image for which sound effects need to be generated, is the color mutation response coefficient stored in the database, is the texture perturbation ratio of a frame image for which sound effects need to be generated, is the texture disturbance response coefficient stored in the database, is the interaction response coefficient stored in the database, is the maximum color gradient value of a frame image for which sound effects need to be generated. is the color gradient response coefficient stored in the database, is the average brightness value of a frame of image for which sound effects need to be generated, is the image brightness response coefficient stored in the database.

[0074] It should be explained that the color mutation response coefficient stored in the database , texture disturbance response coefficient , interaction response coefficient , color gradient response coefficient , image brightness response coefficient The specific steps to obtain are as follows:

[0075] For the color mutation response coefficient: select the image frames marked as those for which sound effects need to be generated in the sample set, calculate the color histogram difference between each frame image and its adjacent previous frame image, and average the color value change amplitude of each pixel point by channel to obtain the color mutation rate of the current sample image; then statistically calculate the correspondence between the color mutation rate of all samples and the emotional intensity level in their sound effect synthesis, fit their change trend through a nonlinear regression model, and extract the sensitive slope coefficient corresponding to the color mutation variable in the fitting equation as the color mutation response coefficient. The numerical type is floating point, and the precision is retained to three decimal places.

[0076] For the texture perturbation response coefficient: grayscale texture analysis is performed on the event area image data in the sample image, the grayscale gradient around each pixel is extracted, and the texture variance value of the area is calculated as the perturbation index. The numerical mapping relationship between the texture perturbation value of all samples and the rate of emotional fluctuation in the generated sound effect is further statistically analyzed. After fitting a polynomial regression model, the nonlinear response coefficient corresponding to the texture variable is extracted as the texture perturbation response coefficient.

[0077] For the interaction response coefficient: under the premise of knowing the color mutation rate and texture disturbance ratio of the sample image, their product is calculated as the collaborative disturbance index, and then paired analysis is performed with the sound effect rhythm adjustment intensity value corresponding to the frame image in the historical annotation. The logarithmic function is used to fit their synergistic effect trend, and the nonlinear weight of the collaborative product term is extracted as the interaction response coefficient, which is used to control the comprehensive impact of color and texture interaction on the sound effect output rhythm.

[0078] For the color gradient response coefficient: read the color channel gradient of the pixel points in the event area of ​​the sample image, count its maximum gradient value, and fit it with the intensity of the high-frequency changes in the emotion in the corresponding sound effect. Extract the influence of the maximum gradient variable on the intensity of the emotion as the color gradient response coefficient.

[0079] For the image brightness response coefficient: the overall average brightness of the sample image is calculated and fitted with the emotional rise amplitude in the sound effect clip corresponding to the image, and the suppression coefficient of the image brightness on the upper limit of the sound effect emotion is extracted as the image brightness response coefficient.

[0080] In this implementation, by constructing an emotional sound effect regulation index, the system integrates multi-dimensional image emotion-related features such as image brightness mean, color mutation rate, local texture disturbance ratio, and maximum color gradient value, effectively realizing the coupled regulation between image context and sound effect emotion parameters. This index not only considers the impact of static image attributes (such as brightness and color changes) on emotional atmosphere, but also comprehensively considers the stimulating effects of local texture disturbance and color gradient changes on emotional fluctuations, and introduces a color-texture interaction factor. The rhythm, intensity and frequency change trends of sound effects are precisely controlled through response coefficients. Each response coefficient is obtained through large-scale labeled sample fitting, with a clear physical source and data statistical basis, ensuring the interpretability and control accuracy of the calculation results. Through the introduction of this index, dynamic adjustment of sound effects in rhythm, emotional intensity and climax regulation can be achieved, thereby improving the synchronization and immersion of sound effects and visual content in emotional expression, which is significantly better than the traditional static solution that only relies on event categories to generate sound effects.

[0081] Specifically, based on the spatial sound image positioning index and the emotional sound effect adjustment index, the specific steps for matching the sound effect clip of each frame image for which sound effects need to be generated are as follows: read the spatial sound image positioning index and the emotional sound effect adjustment index of each frame image for which sound effects need to be generated, and perform weighted analysis to obtain the sound effect matching index of each frame image for which sound effects need to be generated; judge and analyze the sound effect matching index of each frame image for which sound effects need to be generated with several preset sound effect matching intervals, and each sound effect matching interval corresponds to a sound effect clip; the sound effect clip corresponding to the sound effect matching index in a preset sound effect matching interval is used as the generated sound effect of the image.

[0082] The sound effect matching intervals corresponding to the sound effect clips include but are not limited to the following examples:

[0083] Match the frames with sound effect matching index values ​​in the interval [0.0, 0.4) as "basic background" sound effect clips (such as ambient wind sound and mild vibration sound);

[0084] Frame images with sound effect matching index values ​​in the range [0.4, 0.7) are matched as "medium dynamic" sound effect clips (such as the sound of an object sliding or a slight collision);

[0085] Frame images with sound effect matching index values ​​in the range of [0.7, 1.0] are matched as "high-intensity event" sound effect clips (such as explosions and rapid whistles).

[0086] The sound effect clip resources and their corresponding matching intervals are bound one by one and stored in the sound effect resource index table, and are quickly retrieved and inserted according to the matching results at runtime.

[0087] This implementation constructs a joint sound effect matching mechanism based on a spatial sound image localization index and an emotional sound effect modulation index. This mechanism achieves a comprehensive perception of the spatial characteristics and emotional attributes of event regions within an image, driving the precise matching and invocation of sound effect clips. Compared to traditional schemes that simply map sound effects based on event type, this method introduces the sound effect matching index as a mediating variable. This not only considers the location, offset, and disturbance intensity of the event region within the image, but also incorporates information such as color mutations and texture fluctuations of emotional components, thereby improving the contextual adaptability and expressive accuracy of sound effect synthesis. By setting multiple sound effect matching intervals and binding them to sound effect clips one by one, it achieves progressive sound effect adaptation from "background" to "high-intensity event" with good continuity and differentiation, avoiding abrupt switching or distortion mismatch. Furthermore, sound effect resources are bound and dynamically retrieved within the matching intervals according to an index table, ensuring efficient responsiveness and rapid insertion during real-time sound generation. This significantly improves the fluency and realism of generated sound effects, providing precise support for multi-scene image enhancement and audio-visual synthesis.

[0088] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0089] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for synthesizing audio-visual content synchronized sound effects, characterized in that: The following steps are involved: Acquire several frames of image data to be synthesized, and perform feature extraction on each frame to obtain an image feature set for each frame of image to be synthesized; Input the image feature set of each frame to be synthesized into the pre-trained visual event recognition model, identify the frames of images that need to generate sound effects, and output the corresponding trigger event frame information; Based on the image feature set and the trigger event frame information, the spatial sound image localization index and emotional sound effect adjustment index of each frame of the image to generate the sound effect are analyzed; Based on the spatial sound image localization index and the emotional sound effect adjustment index, the sound effect fragment of each frame of the image to be generated is matched, and the sound effect synthesis processing is performed; The trigger event frame information includes the trigger time, event type and event area image data. The event area image data includes the pixel values ​​and two-dimensional coordinates of several event pixels. The visual event recognition model includes a frame-level feature encoding layer, a time series modeling layer, an attention focusing layer, and an event classification and positioning output layer. The specific steps for identifying several frames of images for which sound effects need to be generated and outputting the corresponding trigger event frame information are as follows: In the frame-level feature encoding layer of the visual event recognition model, the image feature set of each frame to be synthesized is received, and field-by-field encoding is performed to output the image encoding feature vector of the corresponding frame; In the time series modeling layer of the visual event recognition model, the image encoding feature vectors of each frame are arranged in chronological order to construct a time series feature tensor, and the time correlation feature vector of each frame is extracted; In the attention focus layer of the visual event recognition model, the temporal correlation feature vector of each frame is read and the difference score between its feature change rate and that of adjacent frames is calculated. Frames with a difference score higher than a set threshold are extracted as candidate trigger event frames. In the event classification and positioning output layer of the visual event recognition model, the corresponding trigger event frame information of several frames of images for which sound effects need to be generated is analyzed and output based on the image encoding feature vectors of the candidate trigger event frames; The specific formula for calculating the spatial sound image localization index of a frame of image for which sound effects need to be generated is as follows: Among them, KsD, BrQ, ZgL, ZpY, and MjZ are the spatial sound image localization index, edge disturbance intensity, inter-frame optical flow vector modulus, center coordinate offset value, and event region area ratio of a frame image for which sound effects need to be generated, respectively. δ1, δ2, δ3, and δ4 are the edge disturbance response coefficient, optical flow response coefficient, center offset response coefficient, and area ratio response coefficient stored in the database, respectively. The specific formula for calculating the emotional sound effect adjustment index of a frame of image for which sound effects need to be generated is as follows: Among them, QyX, StB, WrB, ZsT, and LjZ are the emotional sound effect adjustment index, color mutation rate, texture disturbance ratio, maximum color gradient value, and image brightness mean of a frame image for which sound effects need to be generated, respectively. λ1, λ2, λ3, λ4, and λ5 are the color mutation response coefficient, texture disturbance response coefficient, interaction response coefficient, color gradient response coefficient, and image brightness response coefficient stored in the database, respectively.

2. The method for synthesizing audio-visual content synchronized sound effects according to claim 1, wherein: Each frame of image data includes pixel values ​​and two-dimensional coordinates of several pixel points, and the image feature set includes image brightness mean, color mutation rate and inter-frame optical flow vector modulus.

3. The method for synthesizing audio-visual content synchronized sound effects according to claim 2, wherein: The specific steps for obtaining the image feature set of each frame of image to be synthesized are as follows: Performing average grayscale processing on each frame of the image to be synthesized to obtain the average brightness of each frame of the image to be synthesized; Performing color space histogram analysis on adjacent frames of images to be synthesized to obtain the color mutation rate of each frame of image to be synthesized; The optical flow estimation and averaging processing is performed on the adjacent frame images to be synthesized to obtain the inter-frame optical flow vector modulus value of each frame image to be synthesized.

4. The method for synthesizing audio-visual content synchronized sound effects according to claim 2, wherein: The specific steps for analyzing the spatial sound image localization index of each frame of the image to generate the sound effect are as follows: Based on the event area image data, analyze the edge disturbance intensity of each frame image for which sound effects need to be generated and the coordinates of the centroid of the event area; Based on the two-dimensional coordinates of several pixel points of each frame image for which sound effects are to be generated, the image center coordinates of the corresponding frame image are analyzed, and the center coordinate offset value is analyzed in combination with the coordinates of the centroid point of the event area of ​​the corresponding frame image; Based on the event area image data, analyze the area ratio of the event area of ​​each frame image for which sound effects need to be generated; The inter-frame optical flow vector modulus of each frame image for which sound effects need to be generated is read, and a comprehensive analysis is performed based on the edge disturbance intensity, center coordinate offset value, and event region area ratio of the corresponding frame image to obtain the spatial sound image localization index of each frame image for which sound effects need to be generated.

5. The method for synthesizing audio-visual content synchronized sound effects according to claim 2, wherein: The specific steps for analyzing the emotional sound effect adjustment index of each frame of the image for which sound effects are to be generated are as follows: Based on the event area image data, analyze the local texture disturbance ratio and maximum color gradient value of each frame image to generate sound effects; The image brightness mean and color mutation rate of each frame image for which sound effects need to be generated are read, and a comprehensive analysis is performed in combination with the texture disturbance ratio and maximum color gradient value of the corresponding frame image to obtain the emotional sound effect adjustment index of each frame image for which sound effects need to be generated.

6. The method for synthesizing audio-visual content synchronized sound effects according to claim 1, wherein: Based on the spatial sound image localization index and the emotional sound effect adjustment index, the specific steps for matching the sound effect clip of each frame of the image to generate the sound effect are as follows: Reading the spatial sound image localization index and the emotional sound effect adjustment index of each frame of the image for which the sound effect is to be generated, and performing weighted analysis to obtain the sound effect matching index of each frame of the image for which the sound effect is to be generated; The sound effect matching index of each frame of the image for which the sound effect is to be generated is judged and analyzed with a number of preset sound effect matching intervals, and each sound effect matching interval corresponds to a sound effect segment; The sound effect segment corresponding to the sound effect matching index being within a preset sound effect matching interval is used as the generated sound effect for the image.

Images