A sound scene regulation method and device for a virtual reality scene and a medium

Abstract

The application discloses a sound scene regulation method and device for a virtual reality scene and a medium, relates to the technical field of sound scene path regulation, and comprises the following steps: collecting original recording, synchronous annotation records and a virtual scene, unifying the time axis of the synchronous annotation records and the original recording, and forming an effective annotation table; sequentially intercepting an initial window and an official audio segment, calculating a phase sequence, and then obtaining a processing sequence; performing interval division on the processing sequence, extracting an interval path, and calculating path length, lateral distance and longitudinal distance, and then generating position encoding corresponding to the official audio segment, and forming a position encoding table by summarizing; calculating a playing base point according to the position encoding table, dividing an activation area of the official audio segment in the same interval, binding the playing base point, interval number, activation area, scene label and phase value with the official audio segment, and generating a regulation table. The application improves the synchronization and immersion of the sound scene and the visual scene.

Description

Technical Field

[0001] This invention relates to the field of soundscape path control technology, and in particular to a soundscape control method, device and medium for virtual reality scenes. Background Technology

[0002] The rapid development of virtual reality (VR) technology has brought users a highly immersive multi-sensory experience, among which precise control of soundscape is a key technical aspect for enhancing presence, realizing spatial interaction, and guiding narrative. In virtual reality scene construction, traditional soundscape processing methods typically rely on simple location-based audio triggering and playback mechanisms, or combine pre-recorded ambient sound with 3D audio models. Specifically, conventional methods focus on locating sound source objects and managing trigger events. By binding sound sources to specific coordinates or triggers in the scene model, corresponding audio playback is activated when the user or interactive object enters a preset spatial range. This type of method forms the foundation of current virtual reality soundscape construction, playing a crucial role in creating basic environmental atmosphere and responding to major interactive events.

[0003] However, existing conventional methods have limitations in their effectiveness when dealing with dynamic, complex virtual soundscapes that require high spatial continuity. On one hand, in simulating the spatial continuity of sound sources, conventional methods typically treat sound sources as static or discrete events with simple motion paths. This makes it difficult to accurately simulate the continuous dynamic process of sound properties changing with path curvature, surface material, and relative orientation to the listener as the sound source moves across a complex 3D scene surface. This can lead to spatial sound image jumps and abrupt sound field transitions. On the other hand, in terms of deep integration of sound events with scene geometry, conventional methods often operate the sound activation and playback logic independently of the scene's detailed surface geometry and topology. They lack detailed modeling of physical characteristics such as sound wave propagation path obstruction, reflection, and propagation along surfaces. This results in a need to improve the geometric and physical consistency between the final soundscape and the visual scene, affecting the integrity of the immersive experience. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a soundscape control method for virtual reality scenes to solve the technical problem of coupling and synchronous control of the spatial trajectory, playback logic and scene geometry and topology depth when a sound source moves continuously on the surface of a complex three-dimensional scene.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a soundscape control method for virtual reality scenes, comprising: acquiring original recordings, synchronous annotation records, and virtual scenes; unifying the synchronous annotation records and original recordings along a timeline to form an effective annotation table; sequentially extracting initial windows and formal audio segments; calculating phase sequences to obtain processing sequences; dividing the processing sequences into intervals; extracting interval paths and calculating path lengths, lateral distances, and longitudinal distances to generate position codes corresponding to formal audio segments, and summarizing them to form a position code table; calculating playback base points based on the position code table; dividing formal audio segments within the same interval into activation areas; binding the playback base points, interval numbers, activation areas, scene labels, and phase values ​​to formal audio segments to generate a control table; dividing the activation areas corresponding to each formal audio segment in the control table into continuous time frames; extracting audio data frame by frame and calculating three-dimensional playback coordinates; determining a three-dimensional playback trajectory sequence based on the shortest distance between the three-dimensional playback coordinates and the interval path, binding it to the audio data, and inputting it into a virtual reality engine to achieve three-dimensional spatial soundscape playback.

[0007] As a preferred embodiment of the soundscape control method for virtual reality scenes described in this invention, the synchronous annotation record includes arrival time, departure time, sampling point location, and scene label; the virtual scene includes a three-dimensional model and geometric information; the geometric information includes surface information of the path and surface information of obstacles.

[0008] As a preferred embodiment of the soundscape control method for virtual reality scenes described in this invention, the steps for obtaining the processing sequence are as follows: Using the start time of the original recording as a unified starting point, the synchronization annotation records and the original recording are placed on the same time axis to obtain a unified time index; each synchronization annotation record is checked for validity, and the synchronization annotation records that pass the validity check are reordered from earliest to latest according to their arrival time to form a valid annotation table; based on the valid annotation table, initial windows are extracted from the original recording and sorted according to time order to form an initial window sequence; each initial window is processed by frame segmentation, and the short-time energy of each frame is calculated to obtain a short-time energy sequence; the initial windows are reviewed to obtain a leading energy sequence, and the 95th percentile value of the leading energy sequence is taken as a local threshold; based on the local... The local threshold is used to perform forward and backward scans on the short-time energy sequence of the initial window to determine the correction start and correction end points, thus obtaining the correction interval. If the correction intervals of two adjacent initial windows overlap in time, the short-time energy is compared frame by frame within the overlapping interval, and the frame with the lowest short-time energy is used as the boundary frame to re-divide the boundaries of adjacent correction intervals. Based on the correction interval, the formal audio segment is extracted from the original recording, and the formal audio segment is segmented and its short-time energy is calculated. Based on the change of short-time energy in the formal audio segment relative to the local threshold, the change point is determined, and the time and phase value of the change point are calculated to obtain the phase sequence. The formal audio segment, sampling point position, scene label, and phase sequence are bound together to form a processing record, and all processing records are sorted in chronological order to obtain the processing sequence.

[0009] As a preferred embodiment of the soundscape control method for virtual reality scenes described in this invention, the validity check includes: the sampling point location having both arrival and departure times, departure time being later than arrival time, time region falling completely within the original recording duration, and having both sampling point location and scene label.

[0010] As a preferred embodiment of the soundscape control method for virtual reality scenes described in this invention, the steps of extracting interval paths and calculating path length, lateral distance, and longitudinal distance are as follows: Read the sampling point positions in the processing sequence one by one, map the sampling point positions to the surface of the three-dimensional model of the virtual scene, determine the three-dimensional virtual coordinates of the corresponding processing record; when the sampling point position is inside the three-dimensional model, move it to the surface of the three-dimensional model along the shortest Euclidean distance; when the sampling point position is outside the three-dimensional model, search for the closest position on the surface of the three-dimensional model as the corresponding three-dimensional virtual coordinates; according to the processing sequence order, and considering whether the scene labels are consistent and whether there is a continuous surface path between adjacent three-dimensional virtual coordinates, divide each processing record into intervals. The process involves generating corresponding interval records, extracting the interval start and end points, and using the shortest continuous surface path from the interval start to the interval end as the interval path. If an interval contains only one 3D virtual coordinate, the 3D virtual coordinate is used as the interval start, interval end, and interval path simultaneously, and these are written into the interval record. The path length is accumulated along the interval path to obtain the interval path length. The shortest distance from each 3D virtual coordinate within the interval to the interval path is calculated to obtain the lateral distance, and the lateral scale of the interval is determined. The 3D virtual coordinates corresponding to the formal audio segment are orthogonally projected onto the interval path to obtain the projection point. The cumulative length from the interval path start point along the path to the projection point is calculated and recorded as the longitudinal distance along the path.

[0011] As a preferred embodiment of the soundscape control method for virtual reality scenes described in this invention, the step of summarizing and forming a position coding table refers to calculating the longitudinal position code and the lateral position code according to the longitudinal distance along the path, the path length of the interval, the lateral distance, and the lateral scale, and obtaining the position code, and summarizing all the position codes to obtain the position coding table; if the path length of the interval is zero, the longitudinal position code is recorded as zero, and if the lateral scale is zero, the lateral position code is recorded as zero.

[0012] As a preferred embodiment of the soundscape control method for virtual reality scenes described in this invention, the steps of binding the playback base point, interval number, activation area, scene label, and phase value with the formal audio segment to generate a control table are as follows: Extract the start and end points of the interval into which the projection point of the three-dimensional virtual coordinates corresponding to the formal audio segment falls, and calculate the unit tangent vector; take the upward direction as the global upward direction, extract the unit vector of the global upward direction, calculate the surface normal vector, and calculate the lateral unit vector based on the surface normal vector and the unit tangent vector; read the position encoding table, and calculate the playback base point based on the lateral position encoding, lateral scale, lateral unit vector, and projection point position coordinates; sort all formal audio segments in the same interval according to the vertical position encoding from smallest to largest, and divide the interval into activation areas; rebind the playback base point, interval number, activation area, scene label, and phase value with the formal audio segment to obtain a control record; sort all control records according to the interval number to obtain a control table.

[0013] As a preferred embodiment of the soundscape control method for virtual reality scenes described in this invention, the steps for achieving three-dimensional spatial soundscape playback are as follows: The activation area corresponding to each formal audio segment in the control table is divided into consecutive equally spaced time frames, and the audio data corresponding to each time frame is extracted; based on the playback base point and position encoding, the three-dimensional playback coordinates corresponding to each time frame are calculated frame by frame; the shortest distance from the three-dimensional playback coordinates to the interval path is calculated; the three-dimensional playback coordinates are projected onto the interval path along the direction of the lateral unit vector to obtain the three-dimensional playback trajectory; the three-dimensional playback trajectories are sorted according to time order to obtain a three-dimensional playback trajectory sequence; the audio data of each time frame is sequentially bound to the corresponding three-dimensional playback trajectory sequence; the three-dimensional playback trajectory sequence is input into the virtual reality engine as a callable trajectory data file, and the virtual reality engine is updated to achieve three-dimensional spatial soundscape playback.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the soundscape control method for virtual reality scenes as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the soundscape control method for virtual reality scenes as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: by extracting interval paths and calculating path length, lateral distance and longitudinal distance, and then generating position codes, the spatiotemporal attributes of audio and the geometric topology of the scene are deeply integrated and quantized; by calculating three-dimensional playback coordinates frame by frame and projecting them to form a three-dimensional playback trajectory sequence, the synchronization and immersion of the sound scene and the visual scene are improved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a soundscape control method for virtual reality scenarios.

[0019] Figure 2 This is a schematic diagram of the composition architecture of the soundscape control method for virtual reality scenes.

[0020] Figure 3 A flowchart for processing sequence generation.

[0021] Figure 4 A flowchart generated for location encoding.

[0022] Figure 5 This is a diagram showing the relationship between longitudinal and lateral distances.

[0023] Figure 6 This is a diagram showing the relationship between vertical and lateral position codes. Detailed Implementation

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0026] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0027] Reference Figures 1-6 As one embodiment of the present invention, this embodiment provides a soundscape control method for virtual reality scenes, comprising the following steps: S1. Acquire the original audio recording, synchronous annotation record, and virtual scene. Unify the synchronous annotation record with the original audio recording on the same timeline and form an effective annotation table. Sequentially extract the initial window and the formal audio segment, calculate the phase sequence, and then obtain the processing sequence.

[0028] A fixed reference point is selected for the test area, a local coordinate benchmark is established, sampling points are set, and the positions of the sampling points are recorded. Sampling personnel carry sampling equipment to each sampling point in sequence, stay for a fixed time to sample, and keep the sampling equipment on at all times until all sampling points are sampled to obtain the original audio recording. During the collection process, the arrival time, departure time, and sampling point position of each sampling point are recorded, as well as the scene label. The arrival time, departure time, sampling point position, and scene label are summarized into a synchronous annotation record. A virtual scene of the test area is collected using a 3D scanning device, and a virtual scene coordinate system is established in the virtual scene based on the local coordinate benchmark.

[0029] It should be noted that scene labels are category identifiers used to identify the spatial environment category to which a sampling point belongs.

[0030] It should be noted that the virtual scene includes a 3D model and geometric information. The 3D model includes path objects and obstacle objects. Path objects represent walkable areas, and obstacle objects represent entities that block movement. Geometric information includes surface information of the path and surface information of the obstacle.

[0031] Specifically, the surface information of the path records the length and width of each path in the virtual scene; the surface information of the obstacle records the specific shape, boundary and position of all the geometry that blocks the movement in the virtual scene.

[0032] Using the start time of the original recording as a unified starting point, the synchronized annotation records and the original recording are placed on the same timeline to obtain a unified time index.

[0033] Furthermore, taking the start time of the original recording as a unified starting point, the arrival and departure times of the synchronized annotation records are all converted into relative times with respect to the start time of the original recording, and the corresponding sampling point positions are marked. The time interval between the arrival and departure times corresponding to each sampling point position is taken as the time range, and the synchronized annotation records and the original recording are placed on the same time axis to obtain a unified time index.

[0034] Each synchronized annotation record is checked for validity. The synchronized annotation records that pass the validity check are then reordered from earliest to latest according to their arrival time to form a valid annotation table.

[0035] Furthermore, each synchronized annotation record is checked individually. This includes verifying that the sampling point location of each record simultaneously has both arrival and departure times; that the departure time is later than the arrival time; that the time range of each record falls entirely within the original recording duration; and that each record simultaneously has both a sampling point location and a scene tag. If any check fails, the failed annotation record is not processed further. The synchronized annotation records that pass the checks are then reordered according to their arrival time from earliest to latest, forming a valid annotation table.

[0036] The arrival and departure times of each synchronized annotation record in the valid annotation table are extracted from the original recording to obtain the initial window. The corresponding sampling point positions and scene labels are bound to the initial window and sorted in chronological order to obtain the initial window sequence.

[0037] Each initial window is processed by frame segmentation, and the short-time energy of each frame is calculated to obtain the short-time energy sequence.

[0038] Furthermore, each initial window is divided into frames with a frame length of 20 milliseconds. Adjacent frames are adjacent, and the start and end times of each frame are recorded. The short-time energy of each frame is calculated based on the frame amplitude, expressed as: ; in, For short-term energy, The frame start time, For frame length, For amplitude, For time.

[0039] It should be noted that the frame length is set to 20 milliseconds because short-time analysis usually requires each frame to remain stable, while not being too long and losing temporal resolution. Twenty milliseconds can provide a stable short-time window, so setting it to 20 milliseconds can maintain higher temporal resolution while keeping it approximately stable.

[0040] The short-time energies of each initial window are sorted in chronological order to obtain a short-time energy sequence.

[0041] By reviewing the initial window, a leading energy sequence is obtained, and then a local threshold is obtained. The short-time energy sequence of the initial window is then scanned forward and backward to determine the correction start point and correction end point, thus obtaining the correction interval.

[0042] Looking back one second from the starting point of the initial window, and using the same framing and short-time energy calculation methods as the initial window, perform framing and short-time energy calculations on the one-second lookback to obtain the preamble energy sequence. If the backward movement is less than one second, extract the energy starting from the original recording point. Take the 95th percentile of the preamble energy sequence as the local threshold.

[0043] It should be noted that reviewing for one second is necessary because it balances statistical stability and response time. The 95th percentile value of the leading energy sequence is used to exclude occasional low-energy fluctuations in the background, retaining most of the true environmental energy, thus ensuring that the local threshold reflects the overall background sound level without being disturbed by extremely low energy values.

[0044] Starting from the first frame of the initial window, the system scans frame by frame, comparing the short-time energy of each frame with a local threshold. The start time of the first frame where the short-time energy exceeds the local threshold is taken as the correction starting point. Starting from the last frame of the initial window, the system scans frame by frame, comparing the short-time energy of each frame with the local threshold. The end time of the first frame where the short-time energy exceeds the local threshold is taken as the correction ending point. The interval between the correction starting point and the correction ending point is defined as the correction interval. The difference between the correction ending point and the correction starting point is calculated to obtain the correction duration. If the short-time energy of all frames in the initial window is not greater than the local threshold, then the start point of the initial window is taken as the correction starting point, and the end point of the initial window is taken as the correction ending point.

[0045] When the correction intervals of two adjacent initial windows overlap in time, the short-time energy is compared frame by frame in the overlapping interval. The frame with the lowest short-time energy is selected as the boundary frame. The end time of the boundary frame is used as the correction endpoint of the previous initial window, and the start time of the boundary frame is used as the correction starting point of the next initial window.

[0046] The formal audio segment is extracted from the original recording based on the correction interval. The formal audio segment is then framed and short-time energy is calculated. Based on the change of short-time energy in the formal audio segment relative to the local threshold, the change point is determined, and the time and phase value of the change point are calculated to obtain the phase sequence.

[0047] Furthermore, based on the correction start and end points of each correction interval, corresponding recordings are re-extracted from the original recording as formal audio segments, and the correction duration is taken as the total duration of the formal audio segment. Using the same framing method and short-time energy calculation method as the initial window, the formal audio segment is re-framed, and the short-time energy of each formal audio segment is calculated frame by frame. The short-time energy of each formal audio segment is compared with a local threshold, and frames that satisfy the condition that "the short-time energy of the previous formal audio segment is not higher than the local threshold, and the short-time energy of the current formal audio segment is higher than the local threshold" are identified as change frames, and the midpoint of the change frame is taken as the change point. The difference between the change point and the start point of the formal audio segment is calculated to obtain the change point time, and the ratio of the change point time to the total duration of the formal audio segment is used to calculate the phase value. The phase values ​​are arranged in chronological order to obtain the phase sequence.

[0048] Following the order in the valid annotation table, the formal audio segment, sampling point position, scene label, and phase sequence corresponding to each synchronous annotation record are extracted sequentially. The formal audio segment, sampling point position, scene label, and phase sequence are bound together to form a processing record. All processing records are then sorted in chronological order to obtain the processing sequence.

[0049] S2. Divide the processed sequence into intervals, extract the interval paths and calculate the path length, lateral distance and longitudinal distance, and then generate the position codes corresponding to the formal audio segments, and summarize them to form a position code table; The sampling point positions in the processing sequence are read one by one, and the sampling point positions are mapped to the surface of the three-dimensional model of the virtual scene to determine the three-dimensional virtual coordinates of the corresponding processing record.

[0050] Furthermore, the sampling point positions in the processing sequence are read one by one and directly written into the virtual scene coordinate system. If the current sampling point position falls on the surface of the 3D model of the virtual scene, the current sampling point position is directly used as the 3D virtual coordinates of the corresponding processing record. If the current sampling point position is inside the 3D model of the virtual scene, it starts from the current sampling point position and moves along the shortest Euclidean distance direction to the surface of the 3D model, and the surface position after the movement is determined as the 3D virtual coordinates of the corresponding processing record. If the current sampling point position is outside the 3D model of the virtual scene, the nearest position to the current sampling point position is searched on the surface of the 3D model, and the nearest position is determined as the 3D virtual coordinates of the corresponding processing record.

[0051] Based on the processing sequence, and considering whether the scene labels are consistent and whether there is a continuous surface path between adjacent 3D virtual coordinates, each processing record is divided into intervals, and corresponding interval records are generated.

[0052] Furthermore, the first processing record in the processing sequence is taken as the starting record of the first interval. The scene labels of the second processing record and the first processing record are read respectively. If the scene label of the second processing record is the same as that of the first processing record, the 3D virtual coordinates of the second processing record and the 3D virtual coordinates of the first processing record are read. The surface of the 3D model is searched to see if there is a continuous surface path between the 3D virtual coordinates of the second processing record and the 3D virtual coordinates of the first processing record. If the scene labels are the same and there is a continuous surface path, the second processing record is merged into the first interval where the first processing record is located. If the scene labels are different or there is no continuous surface path, the first interval is ended and the second processing record is taken as the starting record of the second interval. This process is repeated until all processing records have been divided into intervals.

[0053] It should be noted that if there is a path on the surface of the 3D model that can reach another 3D virtual coordinate from the previous 3D virtual coordinate along the surface, then it is determined that there is a continuous path on the surface between the two 3D virtual coordinates; if there is no path on the surface of the 3D model that can reach another 3D virtual coordinate from the previous 3D virtual coordinate along the surface, then it is determined that there is no continuous path on the surface between the two 3D virtual coordinates.

[0054] Each interval is numbered according to the processing sequence to obtain an interval number. An interval record is written for each interval, which includes the formal audio segment of the interval in the processing sequence, the sampling point position, the scene label, and the phase sequence.

[0055] Extract the start and end points of the interval, and use the shortest continuous surface path from the start to the end point as the interval path. Write the start, end, and path of the interval into the interval record.

[0056] Furthermore, if the interval contains more than one 3D virtual coordinate, the first 3D virtual coordinate in the interval is taken as the interval start point, and the last 3D virtual coordinate in the interval is taken as the interval end point. The shortest continuous surface path from the interval start point to the interval end point is searched on the 3D model surface of the virtual scene as the interval path. If the interval contains only one 3D virtual coordinate, this unique 3D virtual coordinate is used as both the interval start point and the interval end point, and this unique 3D virtual coordinate is used as the interval path. The interval start point, interval end point, and interval path are written into the interval record.

[0057] It should be noted that the interval path is obtained by connecting all continuous surface paths in sequence, and the intersection point between any two continuous surface paths is the endpoint of the interval path.

[0058] The path length of the interval is obtained by accumulating the path length along the interval path. The shortest distance from each three-dimensional virtual coordinate in the interval to the interval path is calculated to obtain the lateral distance, and the lateral scale of the interval is determined.

[0059] Furthermore, along the interval path, the path lengths of all paths are summed to obtain the interval path length. If the interval path is a single location, the interval path length is recorded as 0.

[0060] Read all the three-dimensional virtual coordinates in the interval one by one, calculate the shortest distance from each three-dimensional virtual coordinate to the interval path, and obtain the lateral distance.

[0061] If there are lateral distances greater than 0, then all lateral distances greater than 0 are sorted in ascending order, and the median value is extracted as the lateral scale of the interval; if all lateral distances are 0, then the lateral scale of the interval is recorded as 0.

[0062] The three-dimensional virtual coordinates corresponding to the formal audio segment are orthogonally projected onto the interval path to obtain the projection point, and the coordinates of the projection point are recorded.

[0063] It should be noted that a perpendicular line is drawn from the three-dimensional virtual coordinates corresponding to the formal audio segment to the interval path, and the foot of the perpendicular is taken as the projection point; if the foot of the perpendicular is outside the interval path, the endpoint of the interval path closest to the foot of the perpendicular is taken as the projection point.

[0064] Calculate the cumulative length from the starting point of the interval path along the path to the projection point, and denot it as the longitudinal distance along the path. The location code is calculated based on the longitudinal distance along the path, the path length of the interval, the lateral distance, and the lateral scale. The expression is as follows: ; in, For the vertical position encoding of the formal audio segment, This represents the longitudinal distance along the path. The path length of the interval. Lateral position encoding for formal audio segments, The lateral distance for the formal audio segment. Lateral scale, For position encoding, This is the interval number.

[0065] It should be noted that if the path length of the interval is 0, the longitudinal position code is recorded as 0; if the lateral scale is 0, the lateral position code is recorded as 0.

[0066] It should be noted that the longitudinal distance along the path and the path length of the interval are used to jointly characterize the relative position of the formal audio segment in the direction of the interval path length, while the lateral distance and lateral scale are used to jointly characterize the relative offset of the formal audio segment in the lateral direction of the interval path. Therefore, based on the longitudinal distance along the path, the path length of the interval, the lateral distance, and the lateral scale, the spatial position of the formal audio segment relative to the interval path can be quantitatively characterized, and then the position code can be calculated.

[0067] All location codes are summarized to obtain a location code table.

[0068] To demonstrate the spatial distribution of each formal audio segment relative to the interval path after the processed sequence is divided into intervals, and to intuitively reflect the progression of each formal audio segment along the interval path and its lateral offset relative to the interval path, Figure 5 This demonstrates the relationship between the longitudinal and lateral distances of each formal audio segment within the same interval as the segment number changes. The longitudinal distance (solid blue line) represents the cumulative length along the interval path from the start to the end of the interval at the corresponding position of the formal audio segment, while the lateral distance (dashed orange line) represents the shortest distance from the corresponding position of the formal audio segment to the interval path. These two curves visually illustrate the path progression and lateral distribution characteristics of the formal audio segments along the interval path.

[0069] To demonstrate the quantization results obtained through further calculations based on path length, longitudinal distance, lateral distance, and lateral scale, and to intuitively reflect the encoding relationship of each formal audio segment within its respective interval, this invention illustrates the process of quantizing and representing the spatiotemporal attributes of audio. Figure 6 This demonstrates the correspondence between the longitudinal and lateral position codes of each formal audio segment within the same interval, as the segment number changes. The longitudinal position code (solid blue line) represents the relative position of the formal audio segment along the interval path length, while the lateral position code (dashed orange line) represents the relative offset of the formal audio segment along the interval path laterally. These two curves visually illustrate the coded distribution of the spatial attributes of the formal audio segments after quantization.

[0070] S3. Calculate the playback base point based on the position encoding table, divide the formal audio segments in the same interval into activation areas, bind the playback base point, interval number, activation area, scene label and phase value to the formal audio segments, and generate a control table.

[0071] Using the projection points of the three-dimensional virtual coordinates corresponding to all official audio segments in the unified interval as path points, extract the previous and next path points of the projection points of the three-dimensional virtual coordinates corresponding to the official audio segments, and calculate the unit tangent vector, the expression of which is: ; in, As the unit tangent vector, The coordinates of the next path point. These are the coordinates of the previous path point.

[0072] If the path length of the interval is zero, the interval is determined to be a single-point interval, and the unit tangent vector of the previous interval in the processing sequence is used as the unit tangent vector of the single-point interval; if there is no available interval before the current single-point interval, the unit tangent vector of the next interval is used as the unit tangent vector of the single-point interval; if neither the previous interval nor the next interval exists, the forward direction of the processing record to which the single-point interval belongs in the processing sequence is used as the unit tangent vector of the current interval.

[0073] Taking the upward direction as the global upward direction, and extracting the unit vector of the global upward direction, the surface normal vector is calculated as follows: ; in, For the surface normal vector, It is a unit vector in the global direction. It is a cross product.

[0074] It should be noted that when calculating the surface normal vector, the global top-direction unit vector and the unit tangent vector are compared. If the global top-direction unit vector and the unit tangent vector are not parallel, the calculation is performed normally according to the formula. If the global top-direction unit vector and the unit tangent vector are parallel, the global top-direction unit vector can be slightly offset to ensure the legality of the cross product calculation.

[0075] Specifically, a slight offset refers to shifting the global unit vector, as long as it is not parallel to the unit tangent vector.

[0076] The lateral unit vector is calculated based on the surface normal vector and the unit tangent vector, using the following expression: ; in, It is a lateral unit vector.

[0077] Read the position encoding table, and calculate the candidate playback base point based on the lateral position encoding, lateral scale, lateral unit vector, and projection point position coordinates. The expression is: ; in, As a candidate playback base point, These are the coordinates of the projection point.

[0078] It should be noted that the projection point position coordinates represent the corresponding position of the formal audio segment on the interval path, the lateral unit vector represents the lateral direction of the interval path, the lateral position code is used to represent the relative offset of the formal audio segment in the lateral direction of the interval path, and the lateral scale is used to determine the actual offset corresponding to the relative offset. Therefore, by moving along the direction of the lateral unit vector according to the offset determined by the lateral position code and the lateral scale, based on the projection point position coordinates, the candidate playback base point can be obtained.

[0079] Check whether the candidate playback base point overlaps with the obstacle surface. If the candidate playback base point does not overlap with the obstacle surface, then the candidate playback base point is used as the playback base point. If the candidate playback base point overlaps with the obstacle surface, then the candidate playback base point is moved along the direction of the projection point position coordinates until the candidate playback base point does not overlap with the obstacle surface, and the playback base point is obtained.

[0080] All formal audio segments in the same interval are sorted from smallest to largest according to their vertical position encoding, and the interval is divided into activation regions.

[0081] Furthermore, all formal audio segments within the same interval are sorted in ascending order of their vertical position encoding. If the current interval has multiple formal audio segments, the interval is divided into activation regions. The starting point of the interval path is taken as the starting point of the activation region for the first formal audio segment. The average of the vertical distances along the path of the first and second formal audio segments is calculated to obtain the activation vertical distance. The activation vertical distance is extended from the interval starting point to obtain the activation endpoint of the first formal audio segment. The activation endpoint of the first formal audio segment is also taken as the starting point of the activation region for the second formal audio segment. This process is repeated until the interval endpoint of the interval path is taken as the activation endpoint of the last formal audio segment. If the current interval has only one formal audio segment, the activation region starts at the interval path starting point and ends at the interval path ending point.

[0082] The playback base point, interval number, activation area, scene label, and phase value are re-bound to the formal audio segment to obtain the control record.

[0083] It should be noted that when generating control records, an integrity check is performed on the activation area of ​​each control record. By reading the positions of the start and end points of the activation area, it is checked whether the activation area falls completely within the interval path and whether the start point of the activation area is no later than the end point of the activation area. If the activation area exceeds the interval path or the start and end point order is abnormal, the start and end points of the activation area are recalculated to ensure that the activation area is spatially continuous and correct.

[0084] All control records are sorted according to interval numbers to obtain the control table.

[0085] S4. Divide the activation area corresponding to each formal audio segment in the control table into continuous time frames, extract audio data frame by frame and calculate the three-dimensional playback coordinates, determine the three-dimensional playback trajectory sequence based on the shortest distance between the three-dimensional playback coordinates and the interval path, bind it with the audio data, input it into the virtual reality engine, and realize the playback of three-dimensional spatial soundscape.

[0086] For each formal audio segment in the control table, the corresponding activation area is divided into consecutive equally spaced time frames, and the corresponding audio data is obtained. For each time frame, three-dimensional playback coordinates are generated based on the playback base point and position encoding, with the expression as follows: ; in, For 3D playback coordinates, This is the phase value.

[0087] It should be noted that the duration of the equally spaced time frames is between 10 milliseconds and 50 milliseconds, preferably 20 milliseconds. If the duration of the equally spaced time frames is less than 10 milliseconds, the number of time frames will be too large; if the duration of the equally spaced time frames is greater than 50 milliseconds, the interval between the three-dimensional playback coordinates of adjacent time frames will be too large, affecting the continuity of the three-dimensional playback trajectory sequence.

[0088] Calculate the shortest distance from the 3D playback coordinates to the interval path. If the shortest distance from the 3D playback coordinates to the interval path is greater than zero, then project the 3D playback coordinates onto the interval path along the direction of the lateral unit vector, and obtain the 3D playback trajectory along the interval path. If the shortest distance from the 3D playback coordinates to the interval path is equal to zero, then directly obtain the 3D playback trajectory along the interval path.

[0089] The 3D playback trajectory is sorted in chronological order to obtain a 3D playback trajectory sequence.

[0090] The audio data of each time frame is sequentially bound to the corresponding 3D playback trajectory sequence. The 3D playback trajectory sequence is then input into the virtual reality engine as a callable trajectory data file, and the virtual reality engine is updated to achieve 3D spatial soundscape playback.

[0091] It should be noted that a virtual reality engine is a software platform used to generate and manage three-dimensional virtual scenes, and to enable real-time interaction between users and the virtual environment, as well as immersive visual and auditory experiences.

[0092] This embodiment also provides a computer device applicable to a soundscape control method for virtual reality scenes, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the soundscape control method for virtual reality scenes as proposed in the above embodiment.

[0093] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0094] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the sound and scene control method for virtual reality scenes as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0095] In summary, this invention achieves the effect of deep fusion and quantification of the spatiotemporal attributes of audio with the geometric topology of the scene by extracting interval paths and calculating path length, lateral distance and longitudinal distance, and then generating position codes; and improves the synchronization and immersion of sound and visual scenes by calculating three-dimensional playback coordinates frame by frame and projecting them to form a three-dimensional playback trajectory sequence.

[0096] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for soundscape control in virtual reality scenes, characterized in that, include: The process involves acquiring raw audio recordings, synchronized annotation records, and virtual scenes; unifying the timeline of synchronized annotation records with that of raw audio recordings to form an effective annotation table; sequentially extracting initial windows and formal audio segments; calculating phase sequences; and obtaining the processed sequence. The synchronized annotation record includes arrival time, departure time, sampling point location, and scene label; The virtual scene includes a three-dimensional model and geometric information; The arrival and departure times of each synchronous annotation record in the valid annotation table are extracted from the original recording to obtain the initial window, and then sorted in chronological order to form the initial window sequence. Each initial window is divided into frames, and the start and end times of each frame are recorded. The short-time energy of each frame is calculated based on the frame amplitude, using the following expression: ； in, For short-term energy, The frame start time, For frame length, For amplitude, For time; The short-time energies of each initial window are sorted in chronological order to obtain a short-time energy sequence; By reviewing the initial window, a leader energy sequence is obtained, and the 95th percentile value of the leader energy sequence is taken as the local threshold. Using the same framing method and short-time energy calculation method as the initial window, the formal audio segment is re-framed, and the short-time energy of the formal audio segment is calculated frame by frame. Based on the change of short-time energy in the formal audio segment relative to the local threshold, the change point is determined, the time of the change point is calculated, and the ratio of the change point time to the total duration of the formal audio segment is used to calculate the phase value. The phase values ​​are arranged in chronological order to obtain the phase sequence. The formal audio segments, sampling point locations, scene labels, and phase sequences are bound together to form processing records. All processing records are then sorted in chronological order to obtain a processing sequence. The processing sequence is divided into intervals, the interval paths are extracted and the path length, lateral distance and longitudinal distance are calculated, and then the position codes corresponding to the formal audio segments are generated and summarized into a position code table; The extraction of the interval path and the calculation of the path length, lateral distance, and longitudinal distance are as follows: The sampling point positions in the processing sequence are read one by one, and the sampling point positions are mapped to the surface of the three-dimensional model of the virtual scene to determine the three-dimensional virtual coordinates of the corresponding processing record. Extract the start and end points of the interval, and take the shortest continuous surface path from the start to the end of the interval as the interval path; The path length of the interval is obtained by accumulating the path length along the interval path. The shortest distance from each three-dimensional virtual coordinate in the interval to the interval path is calculated to obtain the lateral distance and determine the lateral scale of the interval. The three-dimensional virtual coordinates corresponding to the formal audio segment are orthogonally projected onto the interval path to obtain the projection point; Calculate the cumulative length from the starting point of the interval path along the path to the projection point, and record it as the longitudinal distance along the path; The playback base point is calculated based on the position coding table. The formal audio segments in the same interval are divided into activation areas. The playback base point, interval number, activation area, scene label and phase value are bound to the formal audio segments to generate a control table. The activation area corresponding to each formal audio segment in the control table is divided into continuous time frames. Audio data is extracted frame by frame and three-dimensional playback coordinates are calculated. The three-dimensional playback trajectory sequence is determined based on the shortest distance between the three-dimensional playback coordinates and the interval path, and then bound to the audio data and input into the virtual reality engine to realize three-dimensional spatial sound scene playback.

2. The soundscape control method for virtual reality scenes as described in claim 1, characterized in that, The geometric information includes surface information of the path and surface information of the obstacles.

3. The soundscape control method for virtual reality scenes as described in claim 2, characterized in that, The steps for sequentially extracting the initial window and the actual audio segment are as follows: Using the start time of the original recording as a unified starting point, the synchronized annotation records and the original recording are placed on the same timeline to obtain a unified time index; Each synchronous annotation record is checked for validity. The synchronous annotation records that pass the validity check are reordered from earliest to latest according to their arrival time to form a valid annotation table. Based on the valid annotation table, extract the initial window from the original recording; Based on the local threshold, the short-time energy sequence of the initial window is scanned forward and backward to determine the correction start point and correction end point, and the correction interval is obtained. If the correction intervals of two adjacent initial windows overlap in time, the short-time energy is compared frame by frame in the overlapping interval, and the frame with the lowest short-time energy is used as the boundary frame to re-divide the boundary of the adjacent correction interval. The official audio segment is extracted from the original recording based on the correction interval.

4. The soundscape control method for virtual reality scenes as described in claim 3, characterized in that, The validity check includes the following: the sampling point location has both arrival and departure times, the departure time is later than the arrival time, the time area falls completely within the original recording duration, and the sampling point location and scene label are present simultaneously.

5. The soundscape control method for virtual reality scenes as described in claim 3, characterized in that, The steps for dividing the processed sequence into intervals are as follows: When the sampling point is located inside the 3D model, it moves to the surface of the 3D model along the shortest Euclidean distance. When the sampling point is located outside the 3D model, it searches the 3D model surface for the position closest to the current sampling point as the corresponding 3D virtual coordinate. Based on the processing sequence, and considering whether the scene labels are consistent and whether there is a continuous surface path between adjacent 3D virtual coordinates, each processing record is divided into intervals, and corresponding interval records are generated. When an interval contains only one three-dimensional virtual coordinate, the three-dimensional virtual coordinate is used as the interval start point, interval end point, and interval path simultaneously, and the interval start point, interval end point, and interval path are written into the interval record.

6. The soundscape control method for virtual reality scenes as described in claim 5, characterized in that, The process of compiling the location coding table refers to calculating the longitudinal location code and the lateral location code based on the longitudinal distance along the path, the path length of the interval, the lateral distance, and the lateral scale, and obtaining the location code. All the location codes are then compiled to obtain the location coding table. If the path length of the interval is zero, the longitudinal position code is recorded as zero; if the lateral scale is zero, the lateral position code is recorded as zero.

7. The soundscape control method for virtual reality scenes as described in claim 6, characterized in that, The steps for binding the playback base point, interval number, activation area, scene label, and phase value with the actual audio segment to generate a control table are as follows: Extract the start and end points of the interval into which the projection points of the three-dimensional virtual coordinates corresponding to the formal audio segment fall, and calculate the unit tangent vector; Take the upward direction as the global upward direction, extract the unit vector of the global upward direction, calculate the surface normal vector, and calculate the lateral unit vector based on the surface normal vector and the unit tangent vector; Read the position encoding table and calculate the playback base point based on the lateral position encoding, lateral scale, lateral unit vector, and projection point position coordinates; All formal audio segments in the same interval are sorted from smallest to largest according to their vertical position encoding, and the interval is divided into activation regions. The playback base point, interval number, activation area, scene label, and phase value are re-bound to the formal audio segment to obtain the control record; All control records are sorted according to interval numbers to obtain the control table.

8. The soundscape control method for virtual reality scenes as described in claim 7, characterized in that, The steps to achieve 3D spatial soundscape playback are as follows: The activation area corresponding to each formal audio segment in the control table is divided into consecutive time frames with equal intervals, and the audio data corresponding to each time frame is extracted. Based on the playback base point and position encoding, calculate the three-dimensional playback coordinates corresponding to each time frame; Calculate the shortest distance from the 3D playback coordinates to the interval path; The 3D playback coordinates are projected onto the interval path along the lateral unit vector direction to obtain the 3D playback trajectory; The 3D playback trajectory is sorted in chronological order to obtain a 3D playback trajectory sequence; The audio data of each time frame is sequentially bound to the corresponding 3D playback trajectory sequence. The 3D playback trajectory sequence is then input into the virtual reality engine as a callable trajectory data file, and the virtual reality engine is updated to achieve 3D spatial soundscape playback.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the soundscape control method for virtual reality scenes as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the soundscape control method for virtual reality scenes as described in any one of claims 1 to 8.

Images