Conference live broadcast method, conference live broadcast device and electronic equipment

Abstract

The invention discloses a conference live broadcast method and device and electronic equipment, and relates to the field of data processing. The method comprises the following steps: acquiring video data; splicing the video data to obtain panoramic video data; acquiring audio data of a conference room; performing sound source positioning on the audio data to obtain spatial position coordinates of each spokesman; generating an audio data stream; performing voice recognition on the audio data stream to obtain a voice text of a spokesman; translating the voice text into a target language to obtain a translated text, and generating a subtitle data stream; performing time synchronization on the panoramic video data stream, the audio data stream and the subtitle data stream to obtain a target audio and video data stream; and sending the target audio and video data stream to VR glasses, so that the VR glasses can perform live broadcast on conference content of the conference room. According to the technical scheme, the recognized voice text is translated into the translated text of the target language, it is ensured that participants of different languages can understand the conference content, and the experience feeling of the conference participants is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a conference live broadcast method, device and electronic equipment. Background Art

[0002] With the rapid development of virtual reality (VR) technology, live conference broadcasting using VR has become an emerging form of conference. This technology allows participants to participate in the conference immersively from anywhere in the world, simply by wearing VR equipment, as if they were there. This method not only improves the accessibility and participation of the conference, but also provides an unprecedented interactive experience for remote communication.

[0003] At present, although VR conference live broadcasting has made significant progress in technology, there are still some key technical defects, especially in dealing with the problem of real-time live broadcasting in a multilingual participation environment. In international business meetings or multinational cooperation projects, participants may come from all over the world and speak different languages. The current VR conference system does not take into account the step of language translation, which may lead to language barriers, affect the accurate transmission of information and the overall effect of the meeting, and make the participants feel poor experience.

[0004] Therefore, there is an urgent need for a conference live broadcast method, device and electronic equipment. Summary of the invention

[0005] The present application provides a conference live broadcast method, device and electronic device, which translate the recognized voice text into a translation text in a target language, ensuring that participants of different languages ​​can understand the conference content and improving the experience of conference participants.

[0006] In a first aspect of the present application, a conference live broadcast method is provided, in which video data of different perspectives of a conference room are obtained; each of the video data is spliced ​​to obtain panoramic video data, and a panoramic video data stream is generated based on the panoramic video data; audio data of the conference room is obtained; sound source positioning is performed on the audio data to obtain spatial position coordinates of each speaker; an audio data stream is generated based on the spatial position coordinates; speech recognition is performed on the audio data stream to obtain the speaker's speech text; the speech text is translated into a target language to obtain a translated text, and a subtitle data stream is generated based on the translated text; the panoramic video data stream, the audio data stream and the subtitle data stream are time synchronized to obtain a target audio and video data stream; the target audio and video data stream is sent to VR glasses so that the VR glasses can broadcast the conference content of the conference room live.

[0007] By adopting the above technical solution, by acquiring video data from different perspectives of the conference room and splicing them, a panoramic video data stream is generated, and the real-time acquisition and transmission of the panoramic view of the conference room is realized. At the same time, by acquiring the audio data of the conference room and locating the sound source, an audio data stream with spatial position information is generated, so that users can perceive the position of the speaker, and the spatial sense and presence of the audio are improved. In addition, by performing speech recognition and translation on the audio data, a subtitle data stream is generated, which facilitates users with different language backgrounds to understand the content of the meeting. Finally, by time synchronizing the panoramic video, audio and subtitle data streams, a synchronized target audio and video data stream is obtained, and it is sent to VR glasses for live broadcast, so that users can watch the meeting in person through VR glasses and get an immersive experience. This solution realizes the panoramic acquisition, spatial audio, multi-language subtitles and VR live broadcast functions of the meeting, greatly improving the realism and interactivity of remote meetings, and providing users with a more realistic and natural viewing experience. This technical solution translates the recognized voice text into translated text in the target language, ensuring that participants of different languages ​​can understand the content of the meeting. It effectively solves the communication barriers of participants of different languages ​​in virtual reality meetings, improves the accessibility and participation experience of the meeting, and enables participants from multiple locations around the world to participate in the same meeting without barriers, greatly improving the efficiency and participation of the meeting and enhancing the experience of meeting participants.

[0008] Optionally, the video data is captured by multiple panoramic cameras, and the stitching of each of the video data to obtain the panoramic video data specifically includes: extracting features from each of the video data to determine common feature points of first video data and second video data, the first video data is captured by a first panoramic camera, the second video data is captured by a second panoramic camera, and the first panoramic camera and the second panoramic camera are any two adjacent panoramic cameras among the multiple panoramic cameras; calculating the correspondence between the common feature points between the first video data and the second video data; determining a mapping relationship between the first video data and the second video data based on the correspondence; and stitching each of the video data based on the mapping relationship to obtain the panoramic video data.

[0009] By adopting the above technical solution, the problem of edge overlap and visual discontinuity when adjacent videos are stitched together is solved. By analyzing the mapping relationship between the first video data and the second video data of adjacent videos, the overlapping area between them is determined, and the overlapping area is subjected to special image fusion and transition processing, so as to smooth the transition boundary when the video is stitched together, and eliminate visual mutations and unnaturalness. This solution can generate more visually coherent and natural panoramic videos, and enhance the user's viewing experience. Especially in scenes with large changes in video content and fast movements, image fusion and transition processing can effectively reduce stitching traces and visual artifacts, and maintain the smoothness and consistency of the picture. This solution fully considers the details of video stitching, improves the overall quality of the panoramic video through optimization processing of local areas, and presents users with a more realistic and natural conference scene.

[0010] Optionally, the respective video data are spliced ​​according to the mapping relationship to obtain panoramic video data, specifically comprising: determining an overlapping area between the first video data and the second video data according to a mapping relationship between the first video data and the second video data; and performing image fusion and transition processing on the overlapping area to obtain the panoramic video data.

[0011] By adopting the above technical solution, the problem of edge overlap and visual discontinuity when adjacent videos are stitched together is solved. By analyzing the mapping relationship between adjacent videos, determining the overlapping area between them, and performing special image fusion and transition processing on the overlapping area, the transition boundary during video stitching is smoothed, eliminating visual mutations and unnaturalness. This solution can generate more visually coherent and natural panoramic videos, improving the user's viewing experience. Especially in scenes with large changes in video content and fast movements, image fusion and transition processing can effectively reduce stitching traces and visual artifacts, and maintain the smoothness and consistency of the picture. This solution fully considers the details of video stitching, improves the overall quality of panoramic videos through optimization processing of local areas, and presents users with a more realistic and natural conference scene.

[0012] Optionally, the audio data is collected by multiple microphones, and the sound source positioning of the audio data to obtain the spatial position coordinates of each speaker specifically includes: acquiring the position information and frequency distribution of each microphone; obtaining a microphone array model based on the position information and the frequency distribution; determining the time difference and energy difference between the sounds of each speaker reaching the microphone array model based on the microphone array model; and calculating the spatial position coordinates of each speaker based on the time difference and the energy difference.

[0013] By adopting the above technical solution, the microphone array model is established by obtaining the position information and frequency distribution of multiple microphones, and the spatial position coordinates of the speaker are calculated by using the time difference and energy difference of the sound reaching the microphone array. This solution utilizes the physical characteristics of sound propagation and the spatial distribution of the microphone array to achieve accurate positioning and tracking of the speaker's spatial position, so that the audio data can carry spatial position information, providing data support for subsequent VR audio rendering. The spatial position coordinates of the speaker obtained by positioning are combined with the panoramic video data to achieve spatial synchronization of sound and picture, allowing users to perceive the real position of the speaker in the VR environment, and enhancing the spatial sense and orientation of the audio. This solution overcomes the limitations of traditional single-microphone recording, and through the collaborative work of the microphone array, it realizes the spatial acquisition and positioning of audio, providing a more realistic audio experience for VR meetings and enhancing the user's sense of presence and immersion.

[0014] Optionally, the time synchronization of the panoramic video data stream, the audio data stream and the subtitle data stream to obtain the target audio and video data stream specifically includes: extracting timestamp information from the panoramic video data stream, the audio data stream and the subtitle data stream; calculating the time offset between the panoramic video data stream, the audio data stream and the subtitle data stream according to the timestamp information; and performing time offset correction on the panoramic video data stream, the audio data stream and the subtitle data stream to obtain the target audio and video data stream.

[0015] By adopting the above technical solution, the time difference problem of panoramic video data stream, audio data stream and subtitle data stream during transmission and processing is solved. By extracting the timestamp information of each data stream, calculating the time offset between them, and performing time offset correction, the alignment and synchronization of different data streams on the time axis are achieved. This solution ensures the time consistency of video, audio and subtitles during playback, avoids the phenomenon of asynchronous picture, sound and text, and improves the user's viewing experience. Especially in the case of uneven network transmission and device processing capabilities, the time synchronization method can adaptively adjust the playback progress of different data streams, maintain the relative synchronization relationship between them, and ensure the coherence and integrity of the conference content. This solution fully considers the needs of heterogeneous data fusion, and realizes the synchronous presentation of multimodal data streams through timestamp alignment and offset correction, providing users with a more coordinated and consistent VR live broadcast service.

[0016] Optionally, the translating the speech text into a target language to obtain a translated text specifically includes: obtaining a user's language preference setting, and determining a target language type according to the language preference setting; automatically segmenting and word-segmenting the speech text to obtain a source language word segmentation sequence; using a preset neural network model to convert the source language word segmentation sequence into a target language word segmentation sequence; and performing morphological restoration and grammatical reorganization on the target language word segmentation sequence to obtain a translated text corresponding to the target language type.

[0017] By adopting the above technical solution, the speech text obtained by speech recognition is converted into the target language according to the user's language preference settings to generate the corresponding translation text. The solution automatically performs sentence segmentation and word segmentation on the speech text, and then maps the source language text to the target language text through a preset neural network model, and performs post-processing of word form restoration and grammar reorganization, and finally generates a translation text that conforms to the grammar and expression habits of the target language type. The solution realizes real-time multilingual translation of conference content, meets the understanding needs of users with different language backgrounds, and greatly expands the audience range of the conference. At the same time, the solution also takes into account the user's language preference, provides personalized translation services, and improves the user experience and satisfaction.

[0018] Optionally, after sending the target audio and video data stream to the VR glasses, the method further includes: receiving viewing control instructions sent by the VR glasses, the viewing control instructions including play, pause and rewind; adjusting the playback progress and playback speed of the target audio and video data stream according to the viewing control instructions to obtain an adjusted target audio and video data stream; and sending the adjusted target audio and video data stream to the VR glasses to enable the user to control the viewing of the live content.

[0019] By adopting the above technical solution, users are allowed to send viewing control instructions through VR glasses to adjust the playback progress and playback speed of live content, thereby realizing interaction and control between users and the live broadcast system. The solution receives and parses the control instructions for play, pause, and rewind sent by users, dynamically adjusts the playback status and progress of the audio and video data stream according to the instruction type and parameters, generates an adjusted data stream consistent with the user's operation, and sends it to VR glasses in real time for rendering and presentation. The solution gives users autonomous control over the live content. Users can flexibly control the playback process of the live broadcast according to their own viewing needs and rhythm, which improves the interactivity and flexibility of the live broadcast.

[0020] In a second aspect of the present application, a conference live broadcast device is provided, the device comprising: an acquisition module and a processing module, wherein: the acquisition module is used to acquire video data of different perspectives of the conference room; the processing module is used to splice each of the video data to obtain panoramic video data, and generate a panoramic video data stream based on the panoramic video data; the acquisition module is also used to acquire audio data of the conference room; the processing module is also used to locate the sound source of the audio data to obtain the spatial position coordinates of each speaker; the processing module is also used to generate an audio data stream based on the spatial position coordinates; the processing module is also used to perform speech recognition on the audio data stream to obtain the speaker's speech text; the processing module is also used to translate the speech text into a target language to obtain a translation text, and generate a subtitle data stream based on the translation text; the processing module is also used to time synchronize the panoramic video data stream, the audio data stream and the subtitle data stream to obtain a target audio and video data stream; the processing module is also used to send the target audio and video data stream to VR glasses, so that the VR glasses can broadcast the conference content of the conference room live.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By acquiring video data from different perspectives of the conference room and stitching them together, a panoramic video data stream is generated, which realizes the real-time acquisition and transmission of the panoramic view of the conference room. At the same time, by acquiring the audio data of the conference room and locating the sound source, an audio data stream with spatial position information is generated, so that users can perceive the position of the speaker, which improves the spatial sense and presence of the audio. In addition, by performing speech recognition and translation on the audio data, a subtitle data stream is generated, which facilitates users with different language backgrounds to understand the content of the meeting. Finally, by time synchronizing the panoramic video, audio and subtitle data streams, the synchronized target audio and video data stream is obtained and sent to VR glasses for live broadcast, so that users can watch the meeting in person through VR glasses and get an immersive experience. This solution realizes the panoramic acquisition, spatial audio, multi-language subtitles and VR live broadcast functions of the meeting, greatly improving the realism and interactivity of remote meetings, and providing users with a more realistic and natural viewing experience. This technical solution translates the recognized voice text into translated text in the target language, ensuring that participants of different languages ​​can understand the content of the meeting. It effectively solves the communication barriers of participants of different languages ​​in virtual reality meetings, improves the accessibility and participation experience of the meeting, and enables participants from multiple locations around the world to participate in the same meeting without barriers, greatly improving the efficiency and participation of the meeting and enhancing the experience of meeting participants.

[0024] 2. Solve the problem of edge overlap and visual discontinuity when adjacent videos are stitched together. By analyzing the mapping relationship between the first video data and the second video data of adjacent videos, the overlapping area between them is determined, and special image fusion and transition processing is performed on the overlapping area to smooth the transition boundary during video stitching and eliminate visual mutations and unnaturalness. This solution can generate more visually coherent and natural panoramic videos, improving the user's viewing experience. Especially in scenes with large changes in video content and fast movements, image fusion and transition processing can effectively reduce stitching traces and visual artifacts, and maintain the smoothness and consistency of the picture. This solution fully considers the details of video stitching, improves the overall quality of panoramic videos through optimization processing of local areas, and presents users with a more realistic and natural conference scene.

[0025] 3. Solve the problem of edge overlap and visual discontinuity when adjacent videos are stitched together. By analyzing the mapping relationship between adjacent videos, determining the overlapping area between them, and performing special image fusion and transition processing on the overlapping area, the transition boundary during video stitching is smoothed, eliminating visual mutations and unnaturalness. This solution can generate more visually coherent and natural panoramic videos, improving the user's viewing experience. Especially in scenes with large changes in video content and fast movements, image fusion and transition processing can effectively reduce stitching traces and visual artifacts, and maintain the smoothness and consistency of the picture. This solution fully considers the details of video stitching, improves the overall quality of panoramic videos through optimization processing of local areas, and presents users with a more realistic and natural conference scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a conference live broadcast method disclosed in an embodiment of the present application; Figure 2 It is a module schematic diagram of a conference live broadcast device disclosed in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0027] Explanation of the reference numerals: 201, acquisition module; 202, processing module; 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0028] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0029] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0030] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0031] This application provides a conference live broadcast method, refer to Figure 1 , Figure 1 1 is a flow chart of a conference live broadcast method provided in an embodiment of the present application. The method is applied to a server and includes steps S101 to S109, which are as follows: Step S101: Obtain video data from different viewing angles in the conference room.

[0032] In step S101, a plurality of panoramic cameras are pre-deployed in the conference room, and the server establishes data connection with the panoramic cameras, and can receive video data streams from each panoramic camera in real time through a network interface.

[0033] Step S102: splicing the various video data to obtain panoramic video data, and generating a panoramic video data stream according to the panoramic video data.

[0034] In step S102, video data is captured by multiple panoramic cameras, and features are extracted from each video data to determine common feature points of first video data and second video data, where the first video data is captured by a first panoramic camera, and the second video data is captured by a second panoramic camera, and the first panoramic camera and the second panoramic camera are any two adjacent panoramic cameras among the multiple panoramic cameras; the correspondence between the common feature points of the first video data and the second video data is calculated; based on the correspondence, a mapping relationship between the first video data and the second video data is determined; based on the mapping relationship, each video data is spliced ​​to obtain panoramic video data.

[0035] Specifically, the server will use the SIFT feature extraction algorithm to extract features from the video data stream collected by each panoramic camera, and extract some significant and distinguishing feature points, such as corners and edges. The server will extract a set of feature points for each frame of video image and record their position, scale, and direction. The server will select two adjacent panoramic cameras, such as the first camera and the second camera, and respectively extract feature points from the video data (first video data and second video data) collected by them. The server will use the RANSAC feature matching algorithm to find the common feature points in the two sets of feature points. Common feature points refer to the same physical points that appear in both the first video data and the second video data, and they have similar image features under different viewing angles. These common feature points represent the correspondence between two different shooting perspectives. After finding the common feature points, the server calculates the correspondence between these common feature points between the two different shooting perspectives. The correspondence relationship represents the corresponding position of the feature points in one shooting perspective in another shooting perspective, and then obtains the geometric transformation relationship between the two different shooting perspectives. Through the correspondence between the common feature points in the two shooting perspectives, the server determines the mapping relationship between the first video data and the second video data corresponding to the two shooting perspectives, wherein the first video data corresponds to one shooting perspective, and the second video data corresponds to another shooting perspective. Specifically, the server uses the RANSAC algorithm to randomly sample a part of the common feature points to construct an initial mapping model, and through continuous optimization and evaluation, find an optimal mapping model so that the common feature points in the two shooting perspectives can overlap as much as possible after the coordinate system transformation. The server performs image transformation and image splicing on the first video data and the second video data according to the mapping relationship in the optimal mapping model. Image transformation refers to the coordinate transformation, rotation, and scaling of the image in one perspective according to the mapping relationship so that it is geometrically aligned with the image in another perspective. Image splicing refers to the fusion of the two transformed images in the overlapping area to eliminate the splicing gap and brightness difference to obtain a seamless spliced ​​image.

[0036] The above process can be regarded as the splicing of the first video data and the second video data. The server will process other adjacent perspectives in turn, such as the video data corresponding to the second camera and the third camera, the video data corresponding to the third camera and the fourth camera, etc., repeating the steps of feature extraction, feature matching, mapping relationship estimation and image splicing, and finally splicing the video data of all perspectives into a complete 360° panoramic video data. Then, the server compresses and encapsulates this panoramic video data into a panoramic video data stream suitable for VR glasses to play.

[0037] In a possible implementation, the individual video data are spliced ​​according to a mapping relationship to obtain panoramic video data, specifically including: determining the overlapping area of ​​the first video data and the second video data according to the mapping relationship between the first video data and the second video data; performing image fusion and transition processing on the overlapping area to obtain a panoramic video data stream.

[0038] Specifically, the server determines the overlapping area of ​​the first video data and the second video data. The overlapping area refers to the part of the scene that is commonly covered in the images of the two perspectives. In the overlapping area, the two perspectives capture the same object or scene. The server can determine the overlapping area through the mapping relationship between the first video data and the second video data. The server applies the mapping relationship to the feature points in the first video data, transforms them into the coordinate system of the second video data, and obtains the transformed feature point coordinates. Then, the server searches for these transformed feature point coordinates in the second video data to determine whether they fall within the valid area of ​​the second video data. If the transformed feature point coordinates of a feature point are within the boundary of the second video data, it means that the image area corresponding to the feature point overlaps between the two perspectives. The server records all overlapping feature points and calculates the overlapping area of ​​the first video data and the second video data based on their coordinates, which is usually expressed as a rectangular or polygonal area. After determining the overlapping area, the server uses a weighted average fusion algorithm to perform image fusion and transition processing on the overlapping area to eliminate the visual discontinuity generated when the two perspectives are spliced, and obtain a natural and smooth transition effect. Specifically, the server first calculates the weight of each pixel in the overlapping area in the first video data and the weight of the second video data. The weight is determined based on the distance from the pixel to the boundary of the overlapping area. The closer the pixel is to the boundary, the smaller its weight is, and the farther the pixel is from the boundary, the larger its weight is. This ensures that pixels close to the boundary are more affected by another perspective, while pixels close to the center are more affected by this perspective. The server then performs a weighted average on each pixel in the overlapping area, multiplies the color values ​​of the corresponding pixels in the two perspective images by their respective weights, and then adds them together to obtain the fused color value. By performing a weighted average fusion on all pixels in the overlapping area, the server can obtain a smooth transition splicing result, so that the first video data and the second video data are naturally connected in the overlapping area, and it looks like a continuous panoramic image. The server splices the fused overlapping area with the non-overlapping areas in the two perspectives to obtain a complete panoramic video frame. The server repeats this process, splicing the video data of each pair of adjacent perspectives, and finally generates a continuous panoramic video data stream.

[0039] Step S103: Acquire audio data of the conference room.

[0040] In step S103, the server acquires audio data from multiple microphones deployed in the conference room. These microphones are connected to the server in a wireless manner and can transmit the collected audio data to the server in real time for processing.

[0041] Step S104: localize the sound source of the audio data to obtain the spatial position coordinates of each speaker.

[0042] In step S104, audio data is collected by multiple microphones to obtain the position information and frequency distribution of each microphone; a microphone array model is obtained based on the position information and frequency distribution; based on the microphone array model, the time difference and energy difference of each speaker's voice reaching the microphone array model are determined; based on the time difference and energy difference, the spatial position coordinates of each speaker are calculated.

[0043] Specifically, the server needs to obtain the position information and frequency distribution of each microphone. The position information of the microphone describes the three-dimensional coordinates (x, y, z) of the microphone in the conference room coordinate system, which can be obtained by manual measurement or automatic calibration. For example, when deploying the microphone array, the staff can use a laser rangefinder or a three-dimensional coordinate meter to measure the position of each microphone relative to the origin of the room, and enter the position coordinates into the configuration file of the server. The frequency distribution of the microphone describes the sensitivity and directivity of the microphone at different frequencies, which can be obtained from the microphone data manual. The server establishes a virtual non-uniform linear array (NULA) microphone array model based on the position information and frequency distribution of the microphone. The microphone array model is a mathematical abstraction of the physical microphone array, which describes the geometric structure, directivity and frequency response of the microphone array. After establishing the microphone array model, the server uses the microphone array model to analyze and process the audio data to realize sound source localization. The basic principle of sound source localization is to use the time difference (TDOA) and energy difference (FDOA) of the sound wave from the sound source to different microphones to estimate the spatial position of the sound source through the triangulation algorithm. In an embodiment of the present application, preferably, the server uses the TDOA algorithm to realize sound source localization. The server first performs time-frequency analysis on the audio signal collected by each microphone to extract the characteristic parameters of the speech, such as fundamental tone and resonance peak. Then, the server uses the cross-correlation function or generalized cross-correlation (GCC) to calculate the time difference when different microphones receive the same speech segment. The server lists a set of overdetermined equations based on the microphone array model and the TDOA value, in which the unknown quantity is the spatial coordinates (x, y, z) of the sound source. Finally, the server uses the least squares method (LS) to solve the overdetermined set of equations to obtain the position coordinates of the sound source, that is, the spatial position coordinates of each speaker.

[0044] For example, suppose an array of four microphones is deployed in a conference room. The server builds a NULA microphone array model based on the position coordinates of the microphones and calculates the direction vector and frequency response function in the model. When two speakers speak successively, the server performs time-frequency analysis on the audio signals collected by the four microphones and extracts the pitch and resonance peak of the speech. Then, the server uses the GCC algorithm to calculate the TDOA values ​​of the two speakers' voices reaching the four microphones and obtains eight time difference equations. Next, the server substitutes the time difference equations into the NULA model and constructs a 6-variable overdetermined equation group, in which the unknown quantities are the three-dimensional coordinates of the two speakers (x1, y1, z1, x2, y2, z2). Finally, the server uses the least squares method (LS) to solve the overdetermined equation group and obtains the spatial position coordinates of the two speakers in the conference room coordinate system.

[0045] Step S105: Generate an audio data stream according to the spatial position coordinates.

[0046] In step S105, the server establishes an independent audio channel for the audio data of each speaker. An audio channel refers to a transmission and processing path for audio data, and each audio channel corresponds to a specific sound source, such as a speaker or an audio object. The server creates an equal number of audio channels according to the number of speakers and the format of the audio data, and allocates a unique identifier and buffer to each audio channel. The server calculates the spatial parameters of each audio channel according to the spatial position coordinates of the speaker. The spatial parameters describe the position, direction and distance of the sound source in three-dimensional space, including the following azimuth, pitch angle, distance and sound level; wherein the azimuth represents the angle of the sound source relative to the listener in the horizontal plane, ranging from 0 to 360 degrees; the pitch angle represents the angle of the sound source relative to the listener in the vertical plane, ranging from -90 to 90 degrees; the distance represents the straight-line distance from the sound source to the listener, in meters. The sound level represents the volume of the sound source, in decibels. The server uses trigonometric functions and vector operations to calculate the azimuth, pitch, distance, and sound level of each audio channel based on the speaker's spatial coordinates (x, y, z) and the listener's position coordinates (i.e., the user's virtual position in the conference room). Next, the server performs spatial processing on the audio data of each audio channel and generates corresponding spatial audio data based on the spatial parameters. After the server performs spatial processing on the audio data of all audio channels, a set of spatialized audio data streams is obtained.

[0047] Step S106: Perform speech recognition on the audio data stream to obtain the speaker's speech text.

[0048] In step S106, the server calls a third-party speech recognition service to perform speech recognition on the audio data stream generated in step S105, and converts the speaker's speech signal into corresponding speech text.

[0049] Step S107: Translate the speech text into the target language to obtain a translated text, and generate a subtitle data stream based on the translated text.

[0050] In step S107, the user's language preference setting is obtained, and the target language type is determined according to the language preference setting; the speech text is automatically segmented and word-segmented to obtain a source language word segmentation sequence; the source language word segmentation sequence is converted into a target language word segmentation sequence using a preset neural network model; the target language word segmentation sequence is de-morphologically restored and grammatically reorganized to obtain a translation text corresponding to the target language type.

[0051] Specifically, the server obtains the user's language preference settings and determines the target language type for translation. The language preference settings are set and modified through the user interface or configuration file of the VR client. The server receives the user's language preference settings from the user interaction end of the VR glasses, and maps the language selected by the user to the corresponding target language type, such as Chinese, English, Japanese, etc., according to the predefined language code table or dictionary. The server automatically performs sentence segmentation and word segmentation processing on the speech text obtained in step S106, and converts the continuous speech text into segmented sentences and word sequences. Automatic sentence segmentation refers to the process of automatically dividing the text into independent sentences or clauses according to punctuation, grammatical rules and semantic information. Word segmentation refers to the process of dividing the continuous character sequence in the sentence into meaningful words according to certain rules and vocabulary. The server uses a deep learning-based method to automatically segment and word segment the speech text to obtain a source language word segmentation sequence, such as an English "word" sequence or a Chinese "word" sequence. The server then uses a preset neural network model to convert the source language word segmentation sequence into a target language word segmentation sequence. The preset neural network model adopts the encoder-decoder architecture and consists of several neural network layers, including a word embedding layer, a recurrent neural network (RNN) layer, and an attention mechanism. The server inputs the source language word segmentation sequence into the encoder and extracts the semantic features through word embedding and RNN operations. Then, the server uses the attention mechanism to gradually generate the target language word segmentation sequence in the decoder according to the semantic features of the source language. In the process of generating the target language word segmentation sequence, the server also needs to perform post-processing operations of morphological restoration and grammatical reorganization to improve the fluency and accuracy of the translation. Morphological restoration refers to the process of restoring the inflectional form of a word to its original form or stem, such as restoring the English "goes" to "go". Grammatical reorganization refers to adjusting the order and form of words according to the grammatical rules of the target language to make them conform to the expression habits of the target language. The server combines the word form dictionary and grammatical knowledge base of the target language to perform morphological restoration and grammatical reorganization on the generated target language word segmentation sequence to obtain a translation text corresponding to the target language type. Finally, the server synchronizes the translation text with the audio data stream to generate a subtitle data stream. The subtitle data stream contains the content of the translated text. The server generates corresponding subtitle blocks according to the time boundaries of the translated text and assigns a unique identifier and timestamp to each subtitle block.

[0052] Step S108: Time synchronization is performed on the panoramic video data stream, the audio data stream and the subtitle data stream to obtain the target audio and video data stream.

[0053] In step S108, timestamp information is extracted from the panoramic video data stream, audio data stream and subtitle data stream; the time offset between the panoramic video data stream, audio data stream and subtitle data stream is calculated based on the timestamp information; time offset correction is performed on the panoramic video data stream, audio data stream and subtitle data stream to obtain the target audio and video data stream.

[0054] Specifically, the server needs to time synchronize the panoramic video data stream generated in step S103, the audio data stream generated in step S105, and the subtitle data stream generated in step S107 to ensure that they can maintain a consistent timing relationship during playback, thereby obtaining a synchronized target audio and video data stream. Time synchronization refers to aligning the time axes of multiple data streams so that they play corresponding data frames at the same time to avoid problems such as mismatching of pictures and sounds, and advance or delay of subtitles. First, the server needs to extract the timestamp information in the panoramic video data stream, audio data stream, and subtitle data stream. A timestamp refers to a time mark attached to each data frame, indicating the acquisition, encoding, or transmission time of the data frame. The server calculates the time offset between the panoramic video data stream, audio data stream, and subtitle data stream based on the extracted timestamp information. The time offset refers to the relative offset or difference between the timestamp sequences of different data streams, usually in milliseconds (ms) or microseconds (us). Since different data streams may use different clock sources, sampling rates, frame rates, and other parameters, their timestamp sequences may have a fixed or variable offset relationship. The server performs time offset correction on the panoramic video data stream, audio data stream and subtitle data stream to align their timestamp sequences on the same timeline. The server can select one of the data streams (such as the video data stream) as the reference timeline, and perform offset and scaling transformation on the timestamps of other data streams to keep them consistent with the reference timeline. Finally, the server packages the time-synchronized panoramic video data stream, audio data stream and subtitle data stream into a target audio and video data stream.

[0055] Step S109: Send the target audio and video data stream to the VR glasses so that the VR glasses can broadcast the meeting content in the conference room live.

[0056] In step S109, the server needs to send the target audio and video data stream generated in step S108 to the VR glasses, so that the VR glasses can broadcast and render the meeting content of the virtual conference room in real time, providing users with an immersive VR conference experience.

[0057] After step S109, the method further includes: receiving viewing control instructions sent by the VR glasses, the viewing control instructions including play, pause and rewind; adjusting the playback progress and playback speed of the target audio and video data stream according to the viewing control instructions to obtain an adjusted target audio and video data stream; and sending the adjusted target audio and video data stream to the VR glasses to enable the user to control the viewing of the live content.

[0058] Specifically, the server will also receive the viewing control instructions sent by the VR glasses, and adjust the playback progress and speed of the target audio and video data stream according to the control instructions, so as to realize the user's free control over the live content. The viewing control instructions refer to the interactive commands issued by the user through the VR glasses, such as play, pause, and rewind, which are used to control the playback status and progress of the live video. The server processes and adjusts the target audio and video data stream in real time according to the viewing control instructions, generates the adjusted target audio and video data stream consistent with the user operation, and sends it to the VR glasses to provide a personalized viewing experience. First, the server needs to establish a two-way command channel with the VR glasses to receive the viewing control instructions sent by the VR glasses. The command channel can reuse the audio and video data channel or use a separate control protocol, such as HTTP. The VR glasses generate corresponding viewing control instructions according to the user's operation (such as buttons, gestures, voice, etc.), and send them to the server through the command channel. The viewing control instructions usually include operation types (such as play, pause, fast forward, rewind) and operation parameters (such as the time offset of rewind). After receiving the viewing control instructions, the server processes and adjusts the target audio and video data stream accordingly according to the operation type and parameters of the viewing control instructions. The server adopts different processing strategies for different types of viewing control instructions: For the "play" command, the server resumes sending the target audio and video data stream and determines the starting position of the playback based on the timestamp or frame number specified in the command.

[0059] For the "pause" command, the server pauses the sending of the target audio and video data stream and records the timestamp or frame number when it was paused to facilitate the subsequent resumption of playback.

[0060] For the "rewind" command, the server calculates the new playback position of the target audio and video data stream according to the time offset specified in the command, and starts sending data packets from this position. The server may need to read historical audio and video data from the buffer or storage to support the rewind operation.

[0061] After the server generates the adjusted target audio and video data stream, it sends it to the VR glasses and updates the synchronization information such as timestamp and sequence number in the data packet. After receiving the adjusted target audio and video data stream, the VR glasses decode and render the audio and video data according to the synchronization information to present a playback effect consistent with the user operation.

[0062] Reference Figure 2 The present application also provides a conference live broadcast device, which is a server, and the server includes an acquisition module 201 and a processing module 202, wherein: the acquisition module 201 is used to acquire video data of different perspectives of the conference room; the processing module 202 is used to splice the various video data to obtain panoramic video data, and generate a panoramic video data stream based on the panoramic video data; the acquisition module 201 is also used to acquire audio data of the conference room; the processing module 202 is also used to locate the sound source of the audio data to obtain the spatial position coordinates of each speaker; the processing module 202 is also used to generate an audio data stream based on the spatial position coordinates; the processing module 202 is also used to perform speech recognition on the audio data stream to obtain the speaker's speech text; the processing module 202 is also used to translate the speech text into a target language to obtain a translated text, and generate a subtitle data stream based on the translated text; the processing module 202 is also used to time synchronize the panoramic video data stream, the audio data stream and the subtitle data stream to obtain a target audio and video data stream; the processing module 202 is also used to send the target audio and video data stream to the VR glasses, so that the VR glasses can broadcast the conference content of the conference room live.

[0063] In a possible implementation, video data is captured by multiple panoramic cameras, and each video data is stitched together to obtain panoramic video data, specifically including: a processing module 202 extracts features from each video data to determine common feature points of first video data and second video data, the first video data is captured by a first panoramic camera, and the second video data is captured by a second panoramic camera, and the first panoramic camera and the second panoramic camera are any two adjacent panoramic cameras among the multiple panoramic cameras; the processing module 202 calculates the correspondence between the common feature points of the first video data and the second video data; the processing module 202 determines the mapping relationship between the first video data and the second video data based on the corresponding relationship; the processing module 202 stitches each video data based on the mapping relationship to obtain panoramic video data.

[0064] In a possible implementation, the processing module 202 splices the various video data according to a mapping relationship to obtain panoramic video data, which specifically includes: the processing module 202 determines the overlapping area of ​​the first video data and the second video data according to the mapping relationship between the first video data and the second video data; the processing module 202 performs image fusion and transition processing on the overlapping area to obtain panoramic video data.

[0065] In one possible implementation, audio data is collected by multiple microphones, and the processing module 202 locates the sound source of the audio data to obtain the spatial position coordinates of each speaker, specifically including: the acquisition module 201 obtains the position information and frequency distribution of each microphone; the processing module 202 obtains the microphone array model based on the position information and frequency distribution; the processing module 202 determines the time difference and energy difference between the sounds of each speaker reaching the microphone array model based on the microphone array model; the processing module 202 calculates the spatial position coordinates of each speaker based on the time difference and energy difference.

[0066] In a possible implementation, the processing module 202 performs time synchronization on the panoramic video data stream, the audio data stream and the subtitle data stream to obtain the target audio and video data stream, specifically including: the processing module 202 extracts the timestamp information in the panoramic video data stream, the audio data stream and the subtitle data stream; the processing module 202 calculates the time offset between the panoramic video data stream, the audio data stream and the subtitle data stream according to the timestamp information; the processing module 202 performs time offset correction on the panoramic video data stream, the audio data stream and the subtitle data stream to obtain the target audio and video data stream.

[0067] In a possible implementation, the processing module 202 translates the speech text into a target language to obtain a translated text, specifically including: the acquisition module 201 acquires the user's language preference settings, and determines the target language type according to the language preference settings; the processing module 202 automatically performs sentence segmentation and word segmentation on the speech text to obtain a source language word segmentation sequence; the processing module 202 uses a preset neural network model to convert the source language word segmentation sequence into a target language word segmentation sequence; the processing module 202 performs morphological restoration and grammatical reorganization on the target language word segmentation sequence to obtain a translated text corresponding to the target language type.

[0068] In a possible implementation, after the processing module 202 sends the target audio and video data stream to the VR glasses, the method further includes: the processing module 202 receives viewing control instructions sent by the VR glasses, the viewing control instructions including play, pause and rewind; the processing module 202 adjusts the playback progress and playback speed of the target audio and video data stream according to the viewing control instructions to obtain an adjusted target audio and video data stream; the processing module 202 sends the adjusted target audio and video data stream to the VR glasses to enable the user to control the viewing of the live content.

[0069] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0070] The present application also provides an electronic device. Figure 3 , Figure 3 The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0071] The communication bus 302 is used to realize the connection and communication between these components.

[0072] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0073] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0074] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0075] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a conference live broadcast method.

[0076] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call an application program storing a conference live broadcast method in the memory 305, and when executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0077] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments.

[0078] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0080] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0081] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0083] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0084] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A conference live broadcast method, It is characterized in that The method comprises: Obtain video data from different perspectives in the conference room; splicing the respective video data to obtain panoramic video data, and generating a panoramic video data stream according to the panoramic video data; Acquire audio data of the conference room; Performing sound source positioning on the audio data to obtain the spatial position coordinates of each speaker; generating an audio data stream according to the spatial position coordinates; Performing speech recognition on the audio data stream to obtain a speech text of the speaker; Translating the speech text into a target language to obtain a translation text, and generating a subtitle data stream according to the translation text; Performing time synchronization on the panoramic video data stream, the audio data stream, and the subtitle data stream to obtain a target audio and video data stream; The target audio and video data stream is sent to the VR glasses so that the VR glasses can broadcast the meeting content of the conference room live.

2. The method according to claim 1, It is characterized in that The video data is taken by multiple panoramic cameras, and the video data are stitched together to obtain the panoramic video data, which specifically includes: Extracting features from each of the video data to determine common feature points of the first video data and the second video data, wherein the first video data is shot by a first panoramic camera, and the second video data is shot by a second panoramic camera, and the first panoramic camera and the second panoramic camera are any two adjacent panoramic cameras among the plurality of panoramic cameras; Calculating a correspondence relationship between common feature points of the first video data and the second video data; Determine, according to the corresponding relationship, a mapping relationship between the first video data and the second video data; According to the mapping relationship, each of the video data is spliced ​​to obtain panoramic video data.

3. The method according to claim 2, It is characterized in that The step of splicing the video data according to the mapping relationship to obtain panoramic video data specifically includes: determining an overlapping area between the first video data and the second video data according to a mapping relationship between the first video data and the second video data; The overlapping area is subjected to image fusion and transition processing to obtain the panoramic video data.

4. The method according to claim 1, It is characterized in that The audio data is collected by multiple microphones, and the sound source positioning of the audio data is performed to obtain the spatial position coordinates of each speaker, specifically including: Obtaining the position information and frequency distribution of each of the microphones; Obtaining a microphone array model according to the position information and the frequency distribution; Determining the time difference and energy difference of the voices of each speaker arriving at the microphone array model according to the microphone array model; The spatial position coordinates of each of the speakers are calculated according to the time difference and the energy difference.

5. The method according to claim 1, It is characterized in that The step of performing time synchronization on the panoramic video data stream, the audio data stream, and the subtitle data stream to obtain a target audio and video data stream specifically includes: Extracting timestamp information from the panoramic video data stream, the audio data stream, and the subtitle data stream; Calculating the time offset between the panoramic video data stream, the audio data stream, and the subtitle data stream according to the timestamp information; Time offset correction is performed on the panoramic video data stream, the audio data stream, and the subtitle data stream to obtain the target audio and video data stream.

6. The method according to claim 1, It is characterized in that The translating the speech text into a target language to obtain a translated text specifically includes: Obtaining a user's language preference, and determining a target language type according to the language preference; Automatically segmenting and word-segmenting the speech text to obtain a source language word segmentation sequence; Using a preset neural network model, converting the source language word segmentation sequence into a target language word segmentation sequence; The target language word sequence is de-morphologically restored and grammatically reorganized to obtain a translation text corresponding to the target language type.

7. The method according to claim 1, It is characterized in that After sending the target audio and video data stream to the VR glasses, the method further includes: Receiving a viewing control instruction sent by the VR glasses, wherein the viewing control instruction includes play, pause, and rewind; According to the viewing control instruction, adjusting the playback progress and playback speed of the target audio and video data stream to obtain an adjusted target audio and video data stream; The adjusted target audio and video data stream is sent to the VR glasses to enable the user to control the viewing of the live content.

8. A conference live broadcast device, It is characterized in that The device comprises an acquisition module (201) and a processing module (202), wherein: The acquisition module (201) is used to acquire video data from different viewing angles of the conference room; The processing module (202) is used to splice the various video data to obtain panoramic video data, and generate a panoramic video data stream according to the panoramic video data; The acquisition module (201) is also used to acquire audio data of the conference room; The processing module (202) is further used to perform sound source positioning on the audio data to obtain the spatial position coordinates of each speaker; The processing module (202) is further used to generate an audio data stream according to the spatial position coordinates; The processing module (202) is further used to perform speech recognition on the audio data stream to obtain the speech text of the speaker; The processing module (202) is further used to translate the speech text into a target language to obtain a translation text, and generate a subtitle data stream based on the translation text; The processing module (202) is further used to perform time synchronization on the panoramic video data stream, the audio data stream and the subtitle data stream to obtain a target audio and video data stream; The processing module (202) is also used to send the target audio and video data stream to the VR glasses, so that the VR glasses can broadcast the meeting content of the conference room live.

9. An electronic device, It is characterized in that The electronic device (300) comprises a processor (301), a memory (305), a user interface (303) and a network interface (304), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device (300) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images