Frame insertion compensation method and device for acoustic image anchor point, display equipment and storage medium

Abstract

The invention discloses a frame insertion compensation method and device for an acoustic image anchor point, display equipment and a storage medium, and relates to the technical field of audio signal processing, and the method comprises the steps: obtaining a coordinate point of a target moving object in a video according to a preset frequency; inserting at least one compensation coordinate point in a target track of any two adjacent target coordinate points of each coordinate point; taking each coordinate point and each compensation coordinate point as sound image anchor points of sound signal output of the video; and controlling output parameters of the loudspeaker according to the sound image anchor point, so that the analog output position of the sound signal moves along with the target moving object. The density of the acoustic image anchor points is improved through pin compensation, jumping of the acoustic image anchor points caused by violent position change is avoided in a scene that the position of a moving object in a picture changes violently, the continuity and smoothness of the acoustic image anchor points of the audio signals are improved, the linkage effect between the audio and the video is improved, and the user experience is improved. And the continuity of the whole audio-visual experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio signal processing, and in particular to a method, device, display device and storage medium for interpolation compensation of audio and video anchor points. Background Art

[0002] With the rapid development of multimedia technology, the synchronization and integration of video and audio has become a key factor in improving user experience.

[0003] When a display device plays back the content of a picture, the picture often shows the position of a moving object in the image captured through image recognition, and the continuous movement of a sound-emitting object, such as an airplane flying or an animal running. When the moving object moves, its position in the TV screen may change accordingly. By sampling the video, identifying the coordinates of the moving object in the picture, and assigning a coordinate anchor point (audio-visual anchor point) to the sound information based on the coordinates, the sound information can simulate the effect of being output at the coordinate anchor point, so that the sound effect follows the movement of the moving object.

[0004] However, due to the influence of factors such as the computing power of the main chip and system delay, the sampling frequency of the video screen is often subject to certain restrictions, resulting in the sampling frequency of the video screen being much lower than the processing frame rate of the audio signal. This difference is particularly significant in scenes where the positions of moving objects in the picture change dramatically, making the update of the audio and video anchor points of the audio signal not continuous and smooth enough, thereby weakening the linkage effect between audio and video and reducing the coherence and realism of the overall audio-visual experience.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of the present application is to provide a method, device, display device and storage medium for interpolation compensation of audio and video anchor points, aiming to solve the technical problem of how to improve the linkage effect of audio and video coupling in video.

[0007] To achieve the above purpose, the present application proposes a frame insertion compensation method for an audio-visual anchor point, the method comprising:

[0008] Obtain the coordinate points of the target moving object in the video according to a preset frequency;

[0009] Inserting at least one compensation coordinate point in the target trajectory of any two adjacent target coordinate points of each of the coordinate points;

[0010] Using each of the coordinate points and each of the compensation coordinate points as an audio-visual anchor point for outputting the audio signal of the video, wherein the audio-visual anchor point is a simulated output position of the audio signal of the speaker;

[0011] The output parameters of the speaker are controlled according to the sound image anchor point so that the simulated output position of the sound signal moves along with the target moving object.

[0012] In one embodiment, the step of inserting at least one compensation coordinate point into the target trajectory of any two adjacent target coordinate points of each of the coordinate points comprises:

[0013] Calculate the target trajectory between any two adjacent target coordinate points of each of the coordinate points;

[0014] Acquire the duration between the image frames and the audio frames of the sound signal;

[0015] Calculating a ratio of a duration between the image frames to a duration between the audio frames;

[0016] Determining a target number of compensation coordinate points according to the ratio;

[0017] Insert the target number of compensation coordinate points into the target trajectory.

[0018] In one embodiment, the step of calculating the target trajectory between any two adjacent target coordinate points of each of the coordinate points comprises:

[0019] Obtain any two adjacent target coordinate points from the coordinate points;

[0020] The straight line between the target coordinate points is used as the target trajectory of the target coordinate points.

[0021] In one embodiment, the step of calculating the target trajectory between any two adjacent target coordinate points of each of the coordinate points comprises:

[0022] Acquire any two adjacent target coordinate points among the coordinate points, and a reference coordinate point before or after the target coordinate point;

[0023] Curve fitting is performed on each of the target coordinate points and the reference coordinate point to obtain a target trajectory of the target coordinate point.

[0024] In one embodiment, the step of inserting the target number of compensation coordinate points into the target trajectory includes:

[0025] Calculating a compensation interval according to the number of targets and the duration between image frames;

[0026] Inserting the target number of compensation coordinate points in sequence in the target trajectory according to the compensation interval;

[0027] Or, the target number of compensation coordinate points are sequentially inserted into the target trajectory according to the duration between the audio frames.

[0028] In one embodiment, the step of controlling the output parameters of the speaker according to the audio-visual anchor point comprises:

[0029] Aligning the audio time mark of each of the audio-visual anchor points with the picture time mark of each of the target coordinate points and each of the compensation coordinate points;

[0030] The output parameters of the speaker are controlled according to the timing sequence of the audio time mark and the audio-visual anchor point.

[0031] In one embodiment, the step of acquiring the coordinate points of the target moving object in the video at a preset frequency includes:

[0032] Capturing frames of the video at a preset frequency;

[0033] For each of the picture frames, a target contour of the target moving object in the picture frame is identified, and a center point of the target contour is used as a coordinate point of the target moving object.

[0034] In addition, to achieve the above purpose, the present application also proposes a frame insertion compensation device for an audio-visual anchor point, the device comprising:

[0035] A coordinate point acquisition module is used to acquire the coordinate points of the target moving object in the video according to a preset frequency;

[0036] An interpolation module, used for inserting at least one compensation coordinate point in the target trajectory of any two adjacent target coordinate points of each of the coordinate points;

[0037] A compensation module, used for using each of the coordinate points and each of the compensation coordinate points as an audio-visual anchor point for outputting the audio signal of the video, wherein the audio-visual anchor point is a simulated output position of the audio signal of the speaker;

[0038] The output module is used to control the output parameters of the speaker according to the audio-visual anchor point so that the simulated output position of the sound signal moves with the target moving object.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a display device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the interpolation compensation method of the audio-visual anchor point as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the interpolation compensation method of the audio-visual anchor point as described above are implemented.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the frame insertion compensation method of the audio-visual anchor point as described above are implemented.

[0042] The present application provides a method for interpolation compensation of audio-visual anchor points. The present application identifies the coordinate points of a moving object in a video, then inserts at least one compensation coordinate point in any two target trajectories of each coordinate point, and then uses all the coordinate points and the compensation coordinate points as audio-visual anchor points for outputting the sound signal of the video. During video playback, the output parameters of the speaker are controlled according to the audio-visual anchor points, so that the sound signal output by the speaker follows the movement of the target moving object in the video.

[0043] In summary, the present application performs frame insertion compensation in the coordinate points of the moving objects sampled by the video, so that the sound anchor points of the audio signal are denser than the original video sampling frequency. Compared with the method of determining the sound and image anchor points of the sound signal only by the coordinate points sampled by the video, the present application improves the density of the sound and image anchor points through interpolation compensation. In the scene where the position of the moving object in the picture changes drastically, the jumping of the sound and image anchor points caused by the drastic position change is avoided, and the continuity and smoothness of the sound and image anchor points of the audio signal are improved, thereby improving the linkage effect between audio and video and improving the coherence of the overall audio-visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A flowchart diagram of the first embodiment of the frame insertion compensation method for the audio and video anchor point of the present application;

[0047] Figure 2 A schematic diagram of a straight line trajectory involved in an embodiment of a method for interpolating and compensating an audio and video anchor point of the present application;

[0048] Figure 3 A schematic diagram of a curve track involved in an embodiment of a method for interpolating and compensating an audio and video anchor point of the present application;

[0049] Figure 4 A schematic diagram of interpolation coordinates involved in an embodiment of the interpolation compensation method for an audio-visual anchor point of the present application;

[0050] Figure 5 A schematic diagram of the module structure of the frame insertion compensation device for the audio and video anchor point in the embodiment of the present application;

[0051] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the frame insertion compensation method of the audio and video anchor point in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: obtaining the coordinate points of the target moving object in the video at a preset frequency; inserting at least one compensation coordinate point in the target trajectory of any two adjacent target coordinate points of each of the coordinate points; using each of the coordinate points and each of the compensation coordinate points as the audio-visual anchor points of the sound signal output of the video, wherein the audio-visual anchor points are the simulated output positions of the sound signals of the loudspeakers; and controlling the output parameters of the loudspeakers according to the audio-visual anchor points so that the simulated output position of the sound signal follows the movement of the target moving object.

[0056] In this embodiment, for the convenience of description, the following description is made with the display device as the execution subject.

[0057] With the rapid development of multimedia technology, the synchronization and integration of video and audio has become a key factor in improving user experience.

[0058] When a display device plays back the content of a picture, the picture often shows the position of a moving object in the image captured through image recognition, and the continuous movement of a sound-emitting object, such as an airplane flying or an animal running. When the moving object moves, its position in the TV screen may change accordingly. By sampling the video, identifying the coordinates of the moving object in the picture, and assigning a coordinate anchor point (audio-visual anchor point) to the sound information based on the coordinates, the sound information can simulate the effect of being output at the coordinate anchor point, so that the sound effect follows the movement of the moving object.

[0059] However, due to the influence of factors such as the computing power of the main chip and system delay, the sampling frequency of the video screen is often subject to certain restrictions, resulting in the sampling frequency of the video screen being much lower than the processing frame rate of the audio signal. This difference is particularly significant in scenes where the positions of moving objects in the picture change dramatically, making the update of the audio and video anchor points of the audio signal not continuous and smooth enough, thereby weakening the linkage effect between audio and video and reducing the coherence and realism of the overall audio-visual experience.

[0060] In response to the above problems, the present application provides a frame insertion compensation method for audio and video anchor points. The present application identifies the coordinate points of a moving object in a video, then inserts at least one compensation coordinate point in any two target trajectories of each coordinate point, and then uses all the coordinate points and the compensation coordinate points as audio and video anchor points for outputting the sound signal of the video. During video playback, the output parameters of the speaker are controlled according to the audio and video anchor points, so that the sound signal output by the speaker follows the movement of the target moving object in the video.

[0061] In summary, the present application performs frame insertion compensation in the coordinate points of the moving objects sampled by the video, so that the sound anchor points of the audio signal are denser than the original video sampling frequency. Compared with the method of determining the sound and image anchor points of the sound signal only by the coordinate points sampled by the video, the present application improves the density of the sound and image anchor points through interpolation compensation. In the scene where the position of the moving object in the picture changes drastically, the jumping of the sound and image anchor points caused by the drastic position change is avoided, and the continuity and smoothness of the sound and image anchor points of the audio signal are improved, thereby improving the linkage effect between audio and video and improving the coherence of the overall audio-visual experience.

[0062] It should be noted that the execution subject of this embodiment may be a computing service device, electronic device, display device, etc. with data processing, network communication and program running functions. The following takes the display device as an example to illustrate this embodiment and the following embodiments.

[0063] Based on this, the embodiment of the present application provides a method for interpolating and compensating an audiovisual anchor point, referring to Figure 1 , Figure 1Schematic diagram of the flow chart of the first embodiment of the frame insertion compensation method of the audio and video anchor point of the present application.

[0064] In this embodiment, the interpolation compensation method of the audio-visual anchor point is applied to a display device, and the display device includes: a two-way communication module; the method includes steps S10 to S40:

[0065] Step S10, obtaining the coordinate points of the target moving object in the video according to a preset frequency;

[0066] It should be noted that, in this embodiment, the sampling frequency depends on factors such as the computing power of the main chip of the display device and system delay.

[0067] In this embodiment, one or more target moving objects (such as people, vehicles, etc.) are first defined, and then a preset time interval (i.e., frequency) is used to analyze the video frame to locate the specific position of the target moving object in the video frame, usually expressed in the form of pixel coordinates (x, y).

[0068] Specifically, in a feasible implementation manner, the above step S10 may include steps S11 to S12:

[0069] Step S11, capturing frames of the video at a preset frequency;

[0070] In this embodiment, in order to obtain the coordinate system of the target moving object, it is first necessary to extract individual picture frames from the continuous video stream at specific time intervals (i.e., a preset frequency). The frequency can be set as needed, such as 1 frame per second, 5 frames per second, or 30 frames per second, etc., depending on the processing capability of the system. By intercepting the picture frames, the video can be decomposed into a series of static images, which is convenient for subsequent image processing and analysis.

[0071] Step S12: for each of the picture frames, identifying the target contour of the target moving object in the picture frame, and taking the center point of the target contour as the coordinate point of the target moving object.

[0072] In this embodiment, in each captured picture frame, the tasks of target recognition and target contour extraction need to be performed. Existing image processing technology and computer vision algorithms can be used, such as edge detection, color segmentation, template matching, machine learning models, etc. Target recognition aims to accurately identify the target moving object from the picture frame, while target contour extraction further determines the boundary line or edge of the object.

[0073] Once the target contour is successfully extracted, the center point of the contour can be determined by calculating the average position, centroid or geometric center of all pixels within the contour. This center point is then used as the coordinate point of the target moving object in the frame, usually expressed in the form of pixel coordinates (x, y).

[0074] It is worth noting that since the target moving objects in the video may be deformed or partially occluded due to factors such as occlusion, lighting changes, and perspective changes, in practical applications, more complex and robust algorithms can be used to extract the target contour and calculate the center point to deal with these interferences.

[0075] Step S20, inserting at least one compensation coordinate point into the target trajectory of any two adjacent target coordinate points of each of the coordinate points;

[0076] In this embodiment, in order to smoothly and accurately simulate the moving path of the sound signal of the target moving object, it is necessary to insert additional compensation coordinate points between the original coordinate points. These compensation coordinate points can be calculated by linear interpolation, quadratic interpolation or a more complex trajectory prediction algorithm. The purpose of interpolation is to make up for the possible position gaps between the original coordinate points, so that the final trajectory is smoother and closer to the actual movement situation. The number of inserted compensation coordinate points can be adjusted as needed to achieve the best sound signal simulation effect.

[0077] Step S30, using each of the coordinate points and each of the compensation coordinate points as an audio-visual anchor point for outputting the audio signal of the video, wherein the audio-visual anchor point is a simulated output position of the audio signal of the speaker;

[0078] It should be noted that in this embodiment, when the speakers form an array to output stereo sound, the sound field position can be changed by changing the output power of different speakers, that is, the speaker array can simulate the sound emitted by sound sources at different locations in the space, so that the sound sounds like it is emitted from a certain position in the space, rather than directly from the speaker.

[0079] In this embodiment, all original coordinate points and compensated coordinate points are regarded as audio-visual anchor points to guide the output of sound signals, which means that the output position of the sound signal is no longer fixed, but changes dynamically with the movement of the target moving object. This setting can create a more immersive auditory experience, allowing the audience to feel that the sound seems to come directly from the target moving object in the video.

[0080] Step S40: controlling the output parameters of the speaker according to the audio-visual anchor point so that the simulated output position of the sound signal moves along with the target moving object.

[0081] In this embodiment, it is necessary to dynamically adjust the output parameters of the speaker (such as phase, delay, volume, etc.) according to the position of the audio-visual anchor point to ensure that the simulated output position of the sound signal is consistent with the moving trajectory of the target moving object. By precisely controlling the output parameters of the speaker, the system can create an effect that the sound "follows" the movement of the target moving object, enhancing the audience's viewing experience.

[0082] In a feasible implementation manner, the above step S40 may include steps S41 to S42:

[0083] Step S41, aligning the audio time mark of each of the audio-visual anchor points with the picture time mark of each of the target coordinate points and each of the compensation coordinate points;

[0084] In this embodiment, each audio-visual anchor point has an associated audio timestamp, indicating the position of the anchor point in the audio stream, and each target coordinate point and compensation coordinate point has an associated picture timestamp, indicating the time position of the point in the video stream. Matching the audio time stamp of each audio-visual anchor point with the picture time stamp of the corresponding target coordinate point and compensation coordinate point can ensure that when a specific event occurs in the audio, the output of the speaker can accurately reflect the visual position and time of the event.

[0085] Step S42: controlling the output parameters of the speaker according to the timing sequence of the audio time mark and the audio-visual anchor point.

[0086] In this embodiment, by adjusting the volume, phase, delay and other parameters of the speaker, the desired sound field effect can be created in the space. According to the audio time stamps of the audio and video anchor points and their order in the audio stream, the output of the speaker is controlled in sequence to ensure that the sound effect is synchronized with the playback of the audio and video content.

[0087] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, the above step S20 may include: steps S21 to S25:

[0088] Step S21, calculating the target trajectory between any two adjacent target coordinate points of each of the coordinate points;

[0089] In this embodiment, based on the coordinate points of the target moving object in each picture frame, the straight line or curve trajectory between any two adjacent coordinate points is calculated. The trajectory calculation can be achieved by simple linear interpolation (if the motion is assumed to be uniform) or a more complex trajectory prediction algorithm (if factors such as acceleration and direction change are considered). The target trajectory represents the actual movement path of the target moving object in the video and is the basis for the subsequent insertion of compensation coordinate points.

[0090] Specifically, in a feasible implementation manner, the above step S21 may include steps A10 to A20:

[0091] Step A10, obtaining any two adjacent target coordinate points among the coordinate points;

[0092] In this embodiment, all the acquired coordinate points of the target moving object are traversed first, and then the system selects any two adjacent coordinate points from these coordinate points as processing objects. Adjacent coordinate points refer to two coordinate points that appear consecutively in time, and they respectively represent the positions of the target moving object in two consecutive frames of video.

[0093] Step A20: taking the straight line between the target coordinate points as the target trajectory of the target coordinate points.

[0094] In this embodiment, after any two adjacent target coordinate points are determined, the line between the two coordinate points is regarded as the target trajectory. This simplified processing is usually reasonable when the target moves slowly or the interval between adjacent frames is short, and since the processing method is simpler, the amount of calculation is also small. However, when the target moves quickly or there is a complex motion pattern, a more complex trajectory prediction algorithm may be required to more accurately describe its motion trajectory.

[0095] As an example, see Figure 2 For moving objects such as moving cars whose approximate routes are straight lines, the trajectory can be predicted by directly using the straight line between two points as the target trajectory.

[0096] In another feasible implementation manner, the above step S21 may include steps B10 to B20:

[0097] Step B10, obtaining any two adjacent target coordinate points among the coordinate points, and a reference coordinate point before or after the target coordinate point;

[0098] In this embodiment, the reference coordinate point may be a coordinate point before or after the target coordinate point. The purpose of introducing the reference coordinate point is to consider more context information during curve fitting, thereby obtaining a smoother and more accurate trajectory.

[0099] Step B20, performing curve fitting on each of the target coordinate points and the reference coordinate points to obtain a target trajectory of the target coordinate point.

[0100] In this embodiment, after the adjacent target coordinate points and reference coordinate points are obtained, these points can be used for curve fitting. Curve fitting is a mathematical method used to find a best-fitting curve through a series of data points (coordinate points in this case). This curve will be as close as possible to all the data points while meeting certain smoothness and continuity requirements.

[0101] In the process of curve fitting, different fitting methods can be selected, such as polynomial fitting, spline curve fitting (such as B-spline, NURBS, etc.), least squares method, etc. The specific method to be selected depends on the characteristics of the data, the accuracy requirements of the fitting, and the limitations of computing resources.

[0102] Through curve fitting, the system can obtain a smooth and continuous target trajectory, which not only describes the position change of the target moving object between two adjacent frames, but also takes into account broader movement trends and patterns.

[0103] Through curve fitting, the trajectory of a moving object can be predicted more accurately, making the audio and video matching more accurate.

[0104] As an example, see Figure 3 When targeting moving objects with similar curved motion, such as fired bullets and thrown objects, curve fitting can be used to predict the target trajectory, so that the fitted trajectory is more consistent with the actual motion trajectory in the picture, further improving the synchronization rate of sound and picture.

[0105] Step S22, obtaining the image frame duration between the target coordinate points and the audio frame duration of the sound signal;

[0106] In this embodiment, in order to synchronize the video and audio, it is necessary to know their respective time references. First, the duration between image frames between adjacent target coordinate points is calculated. This is usually obtained by calculating the difference in timestamps of the picture frames where the target coordinate points are located. At the same time, it is also necessary to obtain the duration between audio frames of the sound signal, which depends on the sampling frequency of the audio output.

[0107] Step S23, calculating the ratio of the duration between the image frames to the duration between the audio frames;

[0108] In this embodiment, by comparing the duration between image frames and the duration between audio frames, a ratio between them can be obtained.

[0109] Step S24, determining the target number of compensation coordinate points according to the ratio;

[0110] In this embodiment, based on the calculated ratio, it can be determined how many compensation coordinate points need to be inserted into the target trajectory to adjust the moving speed of the video object so that it matches the playback speed of the audio signal. This number can be calculated by a variety of methods, such as taking into account factors such as the difference between the ratio and 1, the total length of the target trajectory, and the desired synchronization accuracy.

[0111] For example, if the image coordinate frame length is 100ms and the audio frame length is 5ms, the ratio is 20, so 18 new coordinate values ​​need to be inserted between the two points. When the ratio is not an integer, the comparison value can be directly rounded off.

[0112] Step S25, inserting the target number of compensation coordinate points into the target trajectory.

[0113] In this embodiment, finally, according to the determined number of targets, compensation coordinate points are inserted evenly or as needed in the target trajectory, and the positions of these compensation coordinate points are accurately calculated to ensure that they can smoothly connect the original coordinate points and simulate the movement of the target moving object at a new speed (i.e., a speed synchronized with the audio signal). After inserting the compensation coordinate points, the target trajectory will become smoother and more accurate, which will help improve the subsequent audio signal simulation output effect.

[0114] Specifically, in a feasible implementation manner, the above step S25 may include steps S251 to S252:

[0115] Step S251, calculating a compensation interval according to the target quantity and the time length between image frames;

[0116] In this embodiment, in order to evenly insert the compensation points into the target trajectory, the compensation interval can be obtained by dividing the time between image frames by the number of targets plus one.

[0117] Step S252, inserting the target number of compensation coordinate points in sequence in the target trajectory according to the compensation interval;

[0118] In this embodiment, after the compensation interval is determined, we can insert compensation coordinate points between two adjacent original coordinate points along the target trajectory according to the calculated interval.

[0119] Alternatively, in another feasible implementation manner, the above step S25 may further include step S253:

[0120] Step S253: inserting the target number of compensation coordinate points in sequence into the target trajectory according to the duration between the audio frames.

[0121] In this embodiment, in order to make the interval of the compensation coordinate points more closely match the duration between audio frames of the audio signal, the compensation coordinate points may be sequentially inserted into the target track according to the duration between audio frames. For example, in a frame length of 50 ms, if the frame length of the audio signal is 7 ms, the compensation coordinate points may be inserted into the target track at the 7 ms, 14 ms, 21 ms, 28 ms, 35 ms, 42 ms, and 49 ms.

[0122] In another feasible implementation manner, one or more coordinate points may be inserted between adjacent coordinate points by using a median straight line interpolation method.

[0123] For example, please refer to Figure 4 , the coordinates of the first image frame are P1 (X1, Y1), the coordinates of the second image frame are P2 (X2, Y2), the first insertion point is P3 (X3, Y3), and P3 is the midpoint of the line connecting P1 and P2;

[0124] Right now, The third insertion point is P4 (X4, Y4), and the symmetrical position is P5 (X5, Y5), P4 is the midpoint of the line connecting P1 and P3, and P5 is the midpoint of the line connecting P2 and P3;

[0125] The insertion method of P6, P7, P8...P20 is similar...

[0126] Generally, the difference method in the above example is used for odd-numbered differences, and the above algorithm is kept unchanged for even-numbered differences, and the difference closest to P1 in the above example is discarded.

[0127] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the interpolation compensation method of the audio and video anchor points of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0128] This application also provides a frame insertion compensation device for an audio-visual anchor point, please refer to Figure 5 The interpolation compensation device of the audio-visual anchor point includes:

[0129] A coordinate point acquisition module 10 is used to acquire the coordinate points of the target moving object in the video according to a preset frequency;

[0130] An interpolation module 20, configured to insert at least one compensation coordinate point in a target trajectory of any two adjacent target coordinate points of each of the coordinate points;

[0131] A compensation module 30, configured to use each of the coordinate points and each of the compensation coordinate points as an audio-visual anchor point for outputting the audio signal of the video, wherein the audio-visual anchor point is a simulated output position of the audio signal of the speaker;

[0132] The output module 40 is used to control the output parameters of the speaker according to the audio-visual anchor point so that the simulated output position of the sound signal moves along with the target moving object.

[0133] In a feasible implementation manner, the frame insertion module may also be used for:

[0134] Calculate the target trajectory between any two adjacent target coordinate points of each of the coordinate points; obtain the image frame duration between the target coordinate points and the audio frame duration of the sound signal; calculate the ratio of the image frame duration to the audio frame duration; determine the target number of compensation coordinate points based on the ratio; and insert the target number of compensation coordinate points into the target trajectory.

[0135] In a feasible implementation manner, the frame insertion module may also be used for:

[0136] Obtain any two adjacent target coordinate points from among the coordinate points; and use the straight line between the target coordinate points as the target trajectory of the target coordinate points.

[0137] In a feasible implementation manner, the frame insertion module may also be used for:

[0138] Obtain any two adjacent target coordinate points among the coordinate points, and a reference coordinate point before or after the target coordinate point; perform curve fitting on the target coordinate points and the reference coordinate points to obtain the target trajectory of the target coordinate point.

[0139] In a feasible implementation manner, the frame insertion module may also be used for:

[0140] Calculate a compensation interval based on the target number and the duration between image frames; insert the target number of compensation coordinate points in the target trajectory in sequence according to the compensation interval; or, insert the target number of compensation coordinate points in the target trajectory in sequence according to the duration between audio frames.

[0141] In a feasible implementation, the output module may also be used to:

[0142] The audio time mark of each of the audio-visual anchor points is aligned with the picture time mark of each of the target coordinate points and each of the compensation coordinate points; and the output parameters of the speaker are controlled according to the timing sequence of the audio time mark and the audio-visual anchor points.

[0143] In a feasible implementation manner, the coordinate point acquisition module may also be used for:

[0144] The picture frames of the video are captured at a preset frequency; for each of the picture frames, the target contour of the target moving object in the picture frame is identified, and the center point of the target contour is used as the coordinate point of the target moving object.

[0145] The interpolation compensation device for the audio-visual anchor point provided by the present application adopts the interpolation compensation method for the audio-visual anchor point in the above-mentioned embodiment, which can solve the technical problem of how to improve the linkage effect of audio-visual coupling in the video. Compared with the prior art, the beneficial effects of the interpolation compensation device for the audio-visual anchor point provided by the present application are the same as the beneficial effects of the interpolation compensation method for the audio-visual anchor point provided by the above-mentioned embodiment, and other technical features in the interpolation compensation device for the audio-visual anchor point are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0146] The present application provides a display device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the interpolation compensation method of the audio-visual anchor point in the above-mentioned embodiment 1.

[0147] Reference below Figure 6 , which shows a schematic diagram of the structure of a display device suitable for implementing the embodiment of the present application. The display device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, PADs (Portable Application Description: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0148] like Figure 6As shown, the display device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the display device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the display device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a display device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0149] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0150] The display device provided by the present application adopts the interpolation compensation method of the audio-visual anchor point in the above embodiment, which can solve the technical problem of how to improve the linkage effect of audio-visual coupling in the video. Compared with the prior art, the beneficial effects of the display device provided by the present application are the same as the beneficial effects of the interpolation compensation method of the audio-visual anchor point provided by the above embodiment, and the other technical features in the display device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0151] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0152] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0153] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the frame insertion compensation method for the audio-visual anchor point in the above-mentioned embodiment.

[0154] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0155] The computer-readable storage medium may be included in the display device; or may exist independently without being assembled into the display device.

[0156] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a display device, the display device: obtains the coordinate points of a target moving object in a video at a preset frequency; inserts at least one compensation coordinate point in a target trajectory of any two adjacent target coordinate points of each of the coordinate points; uses each of the coordinate points and each of the compensation coordinate points as an audio-visual anchor point for outputting a sound signal of the video, wherein the audio-visual anchor point is a simulated output position of a sound signal of a speaker; and controls the output parameters of the speaker according to the audio-visual anchor point so that the simulated output position of the sound signal follows the movement of the target moving object.

[0157] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0158] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0159] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0160] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned frame insertion compensation method for the audio-visual anchor point, and can solve the technical problem of how to improve the linkage effect of audio-visual coupling in the video. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the frame insertion compensation method for the audio-visual anchor point provided by the above-mentioned embodiment, and will not be repeated here.

[0161] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned frame insertion compensation method for audio and video anchor points when executed by a processor.

[0162] The computer program product provided by the present application can solve the technical problem of how to improve the linkage effect of audio-visual coupling in a video. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the frame insertion compensation method of the audio-visual anchor point provided by the above embodiment, which will not be repeated here.

[0163] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A frame insertion compensation method for an audio-visual anchor point, characterized in that: The method comprises: Obtain the coordinate points of the target moving object in the video according to a preset frequency; Inserting at least one compensation coordinate point in the target trajectory of any two adjacent target coordinate points of each of the coordinate points; Using each of the coordinate points and each of the compensation coordinate points as an audio-visual anchor point for outputting the audio signal of the video, wherein the audio-visual anchor point is a simulated output position of the audio signal of the speaker; The output parameters of the speaker are controlled according to the sound image anchor point so that the simulated output position of the sound signal moves along with the target moving object.

2. The frame insertion compensation method of the audio-visual anchor point according to claim 1, characterized in that: The step of inserting at least one compensation coordinate point into the target trajectory of any two adjacent target coordinate points of each of the coordinate points comprises: Calculate the target trajectory between any two adjacent target coordinate points of each of the coordinate points; Acquire the duration between the image frames and the audio frames of the sound signal; Calculating a ratio of a duration between the image frames to a duration between the audio frames; Determining a target number of compensation coordinate points according to the ratio; Insert the target number of compensation coordinate points into the target trajectory.

3. The frame insertion compensation method of the audio-visual anchor point according to claim 2, characterized in that: The step of calculating the target trajectory between any two adjacent target coordinate points of each of the coordinate points comprises: Obtain any two adjacent target coordinate points from the coordinate points; The straight line between the target coordinate points is used as the target trajectory of the target coordinate points.

4. The frame insertion compensation method of the audio-visual anchor point according to claim 2, characterized in that: The step of calculating the target trajectory between any two adjacent target coordinate points of each of the coordinate points comprises: Acquire any two adjacent target coordinate points among the coordinate points, and a reference coordinate point before or after the target coordinate point; Curve fitting is performed on each of the target coordinate points and the reference coordinate point to obtain a target trajectory of the target coordinate point.

5. The frame insertion compensation method of the audio-visual anchor point according to claim 2, characterized in that: The step of inserting the target number of compensation coordinate points into the target trajectory comprises: Calculating a compensation interval according to the number of targets and the duration between image frames; Inserting the target number of compensation coordinate points in sequence in the target trajectory according to the compensation interval; Or, the target number of compensation coordinate points are sequentially inserted into the target trajectory according to the duration between the audio frames.

6. The frame insertion compensation method of the audio-visual anchor point according to claim 1, characterized in that: The step of controlling the output parameters of the speaker according to the audio-visual anchor point comprises: Aligning the audio time mark of each of the audio-visual anchor points with the picture time mark of each of the target coordinate points and each of the compensation coordinate points; The output parameters of the speaker are controlled according to the timing sequence of the audio time mark and the audio-visual anchor point.

7. The frame insertion compensation method of the audio-visual anchor point according to claim 1, characterized in that: The step of obtaining the coordinate points of the target moving object in the video according to the preset frequency includes: Capturing frames of the video at a preset frequency; For each of the picture frames, a target contour of the target moving object in the picture frame is identified, and a center point of the target contour is used as a coordinate point of the target moving object.

8. A frame insertion compensation device for an audio-visual anchor point, characterized in that: The device comprises: A coordinate point acquisition module is used to acquire the coordinate points of the target moving object in the video according to a preset frequency; An interpolation module, used for inserting at least one compensation coordinate point in the target trajectory of any two adjacent target coordinate points of each of the coordinate points; A compensation module, used for using each of the coordinate points and each of the compensation coordinate points as an audio-visual anchor point for outputting the audio signal of the video, wherein the audio-visual anchor point is a simulated output position of the audio signal of the speaker; The output module is used to control the output parameters of the speaker according to the audio-visual anchor point so that the simulated output position of the sound signal moves with the target moving object.

9. A display device, characterized in that: The display device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the interpolation compensation method for the audio-visual anchor point according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the frame insertion compensation method for the audio-visual anchor point according to any one of claims 1 to 7 are implemented.

Images