Data processing method, electronic device, and computer program product

Abstract

The present disclosure provides a data processing method, an electronic device and a computer program product. The data processing method of the present disclosure comprises: acquiring pose sensor data of a shooting device when collecting a video stream; determining orientation information of the shooting device when collecting the video stream based on the pose sensor data, the orientation information recording a shooting orientation of the shooting device when collecting each video frame of the video stream; and sending the video stream containing the orientation information to a second device, wherein the orientation information is used to drive the second device so that the second device displays an orientation icon representing the shooting orientation when playing the video stream.

Description

Technical Field

[0001] This disclosure relates to the technical fields of data processing, and in particular to a data processing method, electronic device, and computer program product. Background Technology

[0002] With the iteration of mobile internet technology and the maturity of real-time audio and video transmission protocols, live-streaming property viewings has completely reshaped the marketing chain of the real estate brokerage industry. Real estate agents are no longer limited by the physical radius of offline viewings, but have transformed into online "mobile guides," delving into the interior of communities and using a first-person perspective to create an immersive experience for potential buyers on the other end of the screen. This model greatly shortens the psychological distance between property listings and customers, upgrading traditional static text and image displays to dynamic, scenario-based interactions.

[0003] However, the current live streaming format allows viewers to see what the streamer sees on the screen, but they cannot perceive the streamer's absolute location and orientation in physical space. This makes it difficult for viewers to mentally construct the actual spatial layout and geographical relationships of the property. This lack of spatial orientation means that live streaming, which was originally intended to eliminate information asymmetry, may instead increase the cognitive load on viewers due to the ambiguity of location, becoming a key blind spot restricting online property viewing conversion rates. Summary of the Invention

[0004] This disclosure provides a data processing method, an electronic device, and a computer program product.

[0005] According to one aspect of this disclosure, a data processing method is provided, applied to a first device, comprising: acquiring pose sensing data of a shooting device when capturing a video stream; determining orientation information of the shooting device when capturing the video stream based on the pose sensing data, the orientation information recording the shooting orientation of the shooting device when capturing each video frame of the video stream; and sending the video stream containing the orientation information to a second device, wherein the orientation information is used to drive the second device to display an orientation icon representing the shooting orientation when playing the video stream.

[0006] In some embodiments, acquiring pose sensing data of the shooting device when acquiring a video stream includes: using the magnetometer of the shooting device to determine the magnetometer data of the shooting device when acquiring the video stream, the magnetometer data characterizing the three-dimensional magnetic field strength of the shooting device; and using the sensing unit of the shooting device to determine the angular velocity and acceleration of the shooting device when acquiring the video stream, the magnetometer data, the angular velocity, and the acceleration being the pose sensing data.

[0007] In some implementations, determining the orientation information of the capturing device when acquiring the video stream based on the pose sensing data includes: determining the azimuth angle of the capturing device relative to the geographic North Pole when acquiring the video stream based on magnetometer data and acceleration in the pose sensing data; and determining the rotation direction of the capturing device when acquiring the video stream based on the angular velocity, wherein the orientation information includes at least one of the azimuth angle and the rotation direction.

[0008] In some implementations, after determining the orientation information of the capturing device when acquiring the video stream, the process includes: writing the orientation information into an information container of the video stream, the contents of which are transmitted and stored along with the video stream.

[0009] In some implementations, the method further includes: determining a target audio stream suitable for the video stream based on a first audio stream captured by the shooting device and a second audio stream captured by the audio unit of the first device.

[0010] In some implementations, determining a target audio stream suitable for the video stream includes: analyzing the audio content of the first audio stream and the second audio stream, using the audio stream containing human voice signals as the main audio stream and the audio stream without human voice signals as the background audio stream; performing gain boosting on the main audio stream and attenuation on the background audio stream; and fusing the gain-boosted main audio stream and the attenuated background audio stream according to the acquisition time to form the target audio stream.

[0011] In some embodiments, the shooting device is in the form of glasses, and the shooting device has a microphone for receiving audio streams.

[0012] According to another aspect of this disclosure, a data processing method is provided, applied to a second device, comprising: receiving a video stream from a first device, the video stream containing orientation information, the orientation information being determined by the first device based on pose sensing data of a shooting device when acquiring the video stream, the orientation information recording the shooting orientation of the shooting device when acquiring each video frame of the video stream; and displaying an orientation icon representing the shooting orientation when playing the video stream.

[0013] In some implementations, displaying an orientation icon representing the shooting orientation includes: when playing the video stream, controlling the rotation component of the orientation icon to display a rotation trajectory about the shooting orientation according to the acquisition sequence of each video frame in the video stream, wherein the trajectory points of the rotation trajectory are used to indicate the shooting orientation corresponding to the currently playing video frame.

[0014] In some embodiments, the rotating component is a direction dial or a pointer.

[0015] In some embodiments, the orientation icon includes an orientation display control and a rotation direction prompt control, wherein the orientation display control has a rotation component, and the rotation direction prompt control is used to display rotation direction information of the shooting device to the shooting orientation.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a data processing method according to any embodiment of this disclosure.

[0017] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement a data processing method according to any embodiment of this disclosure.

[0018] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method described in any embodiment of this disclosure. Attached Figure Description

[0019] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method according to the embodiments of this disclosure.

[0021] Figure 2 This is a flowchart of a data processing method according to an embodiment of the present disclosure.

[0022] Figure 3 This is a schematic diagram of a multi-terminal interaction process according to an embodiment of the present disclosure.

[0023] Figure 4 This is a flowchart of the orientation information determination process according to an embodiment of the present disclosure.

[0024] Figure 5 This is a schematic diagram illustrating the process of determining the target audio stream according to an embodiment of the present disclosure.

[0025] Figure 6 This is a flowchart of another data processing method according to an embodiment of the present disclosure.

[0026] Figure 7This is a schematic diagram of a display interface according to an embodiment of the present disclosure.

[0027] Figure 8 This is a schematic diagram of another display interface according to an embodiment of the present disclosure.

[0028] Figure 9 This is a schematic block diagram of the structure of a data processing apparatus according to an embodiment of the present disclosure.

[0029] Figure 10 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0030] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0031] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] With the iteration of mobile internet technology and the maturity of real-time audio and video transmission protocols, live-streaming property viewings has completely reshaped the marketing chain of the real estate brokerage industry. Real estate agents are no longer limited by the physical radius of offline viewings, but have transformed into online "mobile guides," delving into the interior of communities and using a first-person perspective to create an immersive experience for potential buyers on the other side of the screen. This model greatly shortens the psychological distance between properties and customers, upgrading traditional static text and image displays to dynamic, scenario-based interactions, allowing homebuyers to transcend time and space limitations and intuitively experience the real-world appearance of properties.

[0033] However, this "first-person" mobile live streaming based on a two-dimensional screen, while breaking down physical boundaries, also constructs a new cognitive barrier. Although viewers can see what the streamer sees through the screen, they cannot perceive the streamer's absolute position and orientation in physical space. When the streamer turns or moves, the image on the screen changes accordingly, but viewers lack a frame of reference and are easily disoriented. This lack of spatial orientation means that live streaming, originally intended to eliminate information asymmetry, may instead increase the cognitive load on viewers due to the ambiguity of location—viewers find it difficult to construct the true spatial layout and geographical relationships of the property in their minds, and therefore cannot determine the building's orientation, lighting conditions, etc.

[0034] Of course, when the host is moving and filming with a handheld device, the constant shaking and instability of the footage will seriously interfere with the viewer's visual experience, not only weakening the presentation of the property details, but also making it difficult to create an immersive viewing atmosphere.

[0035] Therefore, this disclosure proposes a data processing method.

[0036] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method according to embodiments of this disclosure. For example... Figure 1 As shown, in this application scenario, a server 100 and a terminal device 200 may be included. The server 100 and the terminal device 200 can connect via a network or Bluetooth to exchange data. The server 100 can be a cloud server or a physical server, and the terminal device 200 can be a smart device such as a computer, mobile phone, or tablet. The server 100 provides the basic data required to run the data processing method, and the terminal device 200 executes the data processing method of this disclosure based on the basic data provided by the server 100.

[0037] Figure 2 This is a flowchart of a data processing method according to an embodiment of this disclosure. For example... Figure 2 As shown, a data processing method M200 is proposed. By reading the pose sensing data of the shooting device, the orientation information of the shooting device is determined, and the orientation information is sent synchronously when synchronizing the video stream to the second device. This provides data support for the second device to render and display the orientation icon, so that the audience can intuitively obtain the real-time shooting orientation of the shooting device when watching the video stream.

[0038] Step S210: Acquire pose sensing data of the shooting device when capturing video stream.

[0039] The shooting device is used to capture video streams and is equipped with a camera module and a sensor module. In terms of form, the shooting device can be a standalone terminal such as a mobile phone, smart glasses, or action camera, or it can be embedded in a first device with live streaming capabilities (such as the integrated camera module of a mobile phone). There are no restrictions on its form.

[0040] In addition, the shooting equipment has spatial mobility, enabling it to move freely within its space according to the needs of the live streaming scene to capture video streams from different angles. Simultaneously, it undertakes the task of transmitting video streams and pose sensing data, allowing it to transmit the acquired video streams and pose sensing data to the first device in real time.

[0041] The sensor module can include magnetometers, gyroscopes, and accelerometers, which are used to collect magnetometer data, angular velocity, and acceleration, respectively. By configuring the sensor module, the pose sensing data of the shooting device during video stream acquisition can be accurately determined, providing data support for subsequent calculations of the shooting device's orientation information.

[0042] A video stream is essentially a real-time visual signal captured by a camera, containing the object of interest. In a real estate livestream, the video stream can capture any area of ​​the property, such as a panoramic view of the living room, details of a bedroom, or the layout of the kitchen. As the camera rotates and moves in the physical space, the content of the video stream also changes dynamically. Clearly, the content presented in the video stream is not fixed but flexibly adjusted according to the actual needs of the livestream, aiming to provide a comprehensive view of the object of interest and ensure that the visual content highly matches the scene being presented in the livestream.

[0043] Pose sensing data is used to characterize the motion state of the shooting device in its space. This data comprises parameters such as acceleration sequences, angular velocity sequences, and magnetometer data sequences generated in real time during video stream acquisition. As the shooting device rotates and displaces within its space, the acceleration, angular velocity, and magnetometer count values ​​corresponding to each video frame in the video stream dynamically change. This temporal data fluctuation accurately records the motion trajectory of the shooting device, thus providing a physical basis for subsequently determining the real-time shooting orientation of each video frame.

[0044] Step S220: Based on the pose sensing data, determine the orientation information of the shooting device when acquiring the video stream.

[0045] Orientation information records the shooting orientation of the camera device when capturing each video frame of the video stream. It's understandable that the camera device's pose is dynamically changing during the capture of different video frames. Therefore, orientation information is essentially a sequence of shooting orientations, arranged according to the capture sequence of the corresponding video frames.

[0046] Each shooting orientation in the orientation information can include the azimuth angle of the shooting device relative to the geographic North Pole, as well as dynamic parameters such as the rotation direction of the shooting device. By determining the orientation information, data support is provided for reconstructing the device trajectory of the shooting process for the audience, enabling the audience to clearly perceive the geographic orientation and rotation process corresponding to the video stream when watching video frames.

[0047] Step S230: Send the video stream containing orientation information to the second device.

[0048] The second device, serving as the viewer-end terminal for the operational system, boasts highly compatible and diverse hardware forms, encompassing mainstream display devices such as smartphones, tablets, laptops, and smart TVs. The second device renders and plays the video stream forwarded by the first device on the playback interface, and overlays the orientation information from the video stream onto the playback interface as visual icons (i.e., orientation icons), thus presenting viewers with a visual experience of a distant scene that integrates the shooting orientation.

[0049] Orientation information provides the rendering layer of the second device with the orientation information required to render the orientation icon, thus clearly defining the specific direction the orientation icon points on the playback interface. The shooting orientation of the orientation information is synchronized with the acquisition time of each video frame in the video stream; as the video frames are played continuously, the orientation information drives the orientation icon to produce a corresponding dynamic deflection. This real-time linkage between data and vision ensures that viewers can intuitively and accurately perceive the real-time geographical orientation and movement trajectory of the shooting perspective while watching the video stream.

[0050] It should be noted that the data processing method M200 is applied to the first device, which can be... Figure 1 The terminal devices 200, etc. The first device is used as the broadcaster's end to run the business system, and has the ability to acquire video streams through the shooting device and determine its orientation information based on the pose sensing data of the shooting device.

[0051] When the shooting device is embedded in the first device, the first device and the shooting device read the video stream and pose sensing data through an electrical connection. When the shooting device is independent of the first device, the first device and the shooting device are connected via UVC (USB VideoClass, USB video standard), and the first device can read the video stream captured by the shooting device without installing additional drivers; and the shooting device and the first device are connected via Bluetooth or other means to transmit the pose sensing data of the shooting device.

[0052] The first device and the second device exchange data by establishing a network communication link. Specifically, the first device, acting as the streaming end, uploads video stream data encapsulated with orientation information to a streaming media server (e.g., Figure 1 The server 100 shown); the second device acts as the streaming end, obtaining and parsing the video stream from the server via the network, thereby realizing the synchronous presentation and visualization of the video image and orientation information, so that viewers can intuitively obtain the real-time shooting orientation of the shooting device when watching the video stream.

[0053] Figure 3 This is a schematic diagram of a multi-terminal interaction process according to an embodiment of the present disclosure. The following is in conjunction with… Figure 3 This section explains the multi-terminal interaction process involved in implementing the data processing method.

[0054] The implementation of the data processing method involves a shooting device, a first device, and a second device. The shooting device can be a device with a camera module, such as smart glasses. The first device can be a terminal device running the broadcaster's end of the business system, and the second device can be a terminal device running the viewer's end of the business system. Both the first and second devices can be in the form of mobile phones, computers, etc. The interaction methods between the three terminals are described above and will not be repeated here.

[0055] The camera performs steps 301 and 302, namely, acquiring video and audio streams and obtaining pose sensing data. Steps 301 and 302 are usually obtained synchronously because each video frame in the video stream corresponds to a unit of pose sensing data, which mainly records the motion state of the camera while acquiring video frames.

[0056] It should be noted that the audio stream is a sound signal captured synchronously at the same time as the video stream, and its content mainly includes the human voice signals generated by the on-site narration. Based on the synchronous acquisition mechanism, each audio frame in the audio stream corresponds to each video frame in the video stream on the timeline, ensuring accurate timing matching and synchronous presentation of the audiovisual data.

[0057] The first device executes steps 303 to 305, determining the orientation information of the shooting device based on the pose sensing data; embedding the orientation information into the information container of the video stream; performing noise reduction processing on the audio stream to determine the target audio stream.

[0058] As a high-performance intelligent terminal, the first device typically has a processor with computing power exceeding that of the shooting device. It can efficiently fuse pose sensing data and calculate complex orientation information, ensuring the accuracy of the orientation information. In addition, concentrating the computing tasks on the first device avoids adding extra hardware costs and power consumption to the shooting device, and simplifies the structural design of the shooting device, allowing it to focus more on the efficient acquisition of video and audio, thereby achieving the optimal balance between overall system performance and cost.

[0059] The information container for a video stream can be an SEI (Supplemental Enhancement Information). The SEI can carry video stream-related information, such as orientation information, without interfering with the core image decoding and reconstruction of the video stream. By encapsulating the orientation information in the SEI, not only is frame-level synchronous transmission of the orientation information and the video stream achieved, but it also ensures that the second device can accurately determine the orientation information corresponding to the video frame during playback, thereby overlaying and displaying the orientation icon.

[0060] The first device performs noise reduction processing on the acquired audio stream to determine the target audio stream. When the audio stream is only acquired by the camera, steady-state sounds (such as equipment operation noise and wind noise) and transient interference in the environment can be identified and suppressed through methods such as spectral subtraction or adaptive filtering algorithms. By stripping noise components from the mixed signal and retaining the human voice signal, the signal-to-noise ratio of the audio is improved, and a clear and pure target audio stream is finally output, ensuring that viewers can have a high-quality listening experience when watching the live broadcast.

[0061] It should be noted that the audio stream can also be acquired through the first device. That is, the shooting device acquires the first audio stream, and the audio unit of the first device acquires the second audio stream. In this case, it is necessary to first determine the audio stream containing the human voice signal, and then determine the main audio stream and the background audio stream; then, the main audio stream is boosted and the background audio stream is attenuated to finally obtain the target audio stream; of course, this process will be introduced later, and will not be explained in detail here.

[0062] Then, step 306 is executed, whereby the first device sends the target audio stream and video stream to the second device. Next, the second device executes step 307, playing the video stream and the target audio stream, and displaying an orientation icon.

[0063] Of course, when the second device plays the audio stream, it will match and play the corresponding audio stream according to the acquisition sequence of the audio stream to achieve the effect of combining audio and video, providing support for the broadcaster to explain the object of interest and the playback of background sound effects, so that the audience can have an immersive experience while learning about the object of interest.

[0064] In some implementations, acquiring pose sensing data of the shooting device when capturing a video stream includes: using the magnetometer of the shooting device to determine the magnetometer data of the shooting device when capturing the video stream, the magnetometer data representing the three-dimensional magnetic field strength of the shooting device; and using the sensing unit of the shooting device to determine the angular velocity and acceleration of the shooting device when capturing the video stream, the magnetometer data, angular velocity and acceleration being pose sensing data.

[0065] As a sensor measuring the strength and direction of a magnetic field, the magnetometer performs the core function of an "electronic compass." In this solution, the magnetometer is used to collect magnetometer data, which characterizes the three-dimensional magnetic field strength of the imaging device along the X, Y, and Z axes. The X and Y axes form a horizontal coordinate system parallel to the ground plane and are orthogonal to each other; the Z axis is perpendicular to the ground plane. By accurately sensing the vector distribution of the Earth's magnetic field in this coordinate system, the azimuth angle of the imaging device relative to the geographic North Pole can be calculated, effectively correcting the cumulative errors generated by other inertial sensors in the direction calculation process.

[0066] The sensing unit may include a gyroscope and an accelerometer, which work together to capture the dynamic motion characteristics of the device. The gyroscope measures angular velocity, i.e., the speed and direction of the device's rotation around its axis, primarily reflecting changes in the shooting orientation. The accelerometer measures acceleration, i.e., the rate of change of the device's velocity in space, primarily reflecting the device's displacement trend and tilt attitude. Magnetometer data, angular velocity, and acceleration together constitute complete pose sensing data, providing multi-dimensional physical evidence for subsequent accurate calculation of the shooting device's orientation information.

[0067] In some implementations, determining the orientation information of the camera device when acquiring a video stream based on pose sensing data includes: determining the azimuth of the camera device relative to the geographic North Pole when acquiring the video stream based on magnetometer data and acceleration in the pose sensing data; and determining the rotation direction of the camera device when acquiring the video stream based on angular velocity in the pose sensing data, wherein the orientation information includes at least one of the azimuth and rotation direction.

[0068] The acceleration readings from the accelerometer include a gravitational component, thus allowing the direction of gravity to be sensed based on the acceleration, thereby determining the current tilt attitude of the filming device. Then, the magnetometer data is corrected and projected from the "device coordinate system constructed by X, Y, and Z" to the "horizontal geographic coordinate system" to eliminate measurement errors caused by device tilt, ultimately calculating the azimuth angle of the filming device relative to the geographic North Pole. That is, based on the magnetic field strength projected to the geographic North Pole direction in the horizontal geographic coordinate system, the angle between the device axis and the geographic North Pole direction is calculated using the arctangent function, and combined with local magnetic declination correction, finally obtaining the precise azimuth angle relative to the geographic North Pole.

[0069] Angular velocity includes the three-dimensional angular velocity components of the shooting device rotating around the X, Y, and Z axes. By integrating the angular velocity in the time domain, the angular change and rotation direction of the shooting device within the continuous video frame acquisition interval can be accurately calculated. This process can sensitively capture minute attitude adjustments of the shooting device, thereby determining whether the device is rotating left, right, up, or down, and the specific magnitude of the rotation.

[0070] Azimuth provides an absolute geographic spatial reference, mainly used to indicate the "absolute orientation" of the shot (such as due north, southeast, etc.), allowing the second device to overlay azimuth icons when presenting the video stream, helping viewers quickly establish a sense of geographic orientation; while rotation direction describes relative dynamic changes, mainly used to characterize the "camera movement trajectory" of the lens and the smooth transition of the orientation icon rotation component.

[0071] Figure 4 This is a flowchart illustrating the process of determining orientation information according to embodiments of this disclosure. Figure 4 As shown, this disclosure provides a process for determining orientation information.

[0072] In this embodiment, steps 401 to 404 are provided, namely: using the magnetometer of the shooting device to determine the magnetometer data of the shooting device when acquiring the video stream; using the sensing unit of the shooting device to determine the angular velocity and acceleration of the shooting device when acquiring the video stream; based on the magnetometer data and acceleration, determining the azimuth angle of the shooting device relative to the geographic North Pole when acquiring the video stream; and based on the angular velocity, determining the rotation direction of the shooting device when acquiring the video stream.

[0073] This implementation provides only one way to determine orientation information. All the technical means involved in the implementation process have been introduced above and will not be described again here.

[0074] In some implementations, after determining the orientation information of the shooting device when capturing the video stream, the orientation information is written into the information container of the video stream, and the contents of the information container are transmitted and stored along with the video stream.

[0075] The information container can be an SEI, which can encapsulate orientation information as an invisible attachment to the video stream, enabling frame-level synchronous transmission of data and video frames. This does not affect the decoding and display of the video stream, but ensures that the second device can accurately parse and overlay the orientation icon when playing the stream. It is also compatible with various mainstream video decoders, and can achieve seamless transmission and presentation of orientation information without additional development and adaptation.

[0076] This solution supports independent acquisition and flexible adaptation of the first and second audio streams. Specifically, the camera can focus on capturing ambient sound and / or the guide's narration as the first audio stream, while the first device simultaneously captures the presenter's narration or interactive audio as the second audio stream. This separate acquisition mode not only enables independent operation of the camera and narration perspectives but also significantly improves applicability in various scenarios; for example, in guided tour mode, the professional narration captured by the camera is prioritized, while in interactive mode, the presenter's audio from the first device is integrated. Of course, in simplified scenarios such as single-person operation, narration and acquisition can also be completed simultaneously through the camera, achieving efficient matching of audio and video.

[0077] Figure 5 This is a schematic diagram illustrating the process of determining the target audio stream according to an embodiment of this disclosure. Figure 5 The diagram illustrates a process for determining a target audio stream.

[0078] In this embodiment, steps 501 to 506 are provided, namely: analyzing the audio content of the first audio stream and the second audio stream; using the audio stream with human voice signal as the main audio stream and the audio stream without human voice signal as the background audio stream (for example, determining whether the first audio stream has human voice signal; when the first audio stream does not have human voice signal, the second audio stream is used as the main audio stream and the first audio stream is used as the background audio stream; otherwise, the first audio stream is used as the main audio stream and the second audio stream is used as the background audio stream); performing gain boosting processing on the main audio stream and attenuation processing on the background audio stream; and fusing the gain-boosted main audio stream and the attenuated background audio stream according to the acquisition time to form the target audio stream.

[0079] In the specific implementation of gain enhancement and attenuation processing, automatic gain control and dynamic range compression techniques can be combined: For the main audio stream, the human voice signal is first accurately located through voice activity detection, and then the gain coefficient is dynamically adjusted according to the average loudness of the human voice signal to stabilize the human voice signal level within a preset target range. At the same time, slight compression is applied to reduce the dynamic range of the human voice signal, making it clearer and more prominent. For the background audio stream, a fixed attenuation or adaptive attenuation strategy can be adopted. For example, the attenuation amplitude of the background sound can be adjusted inversely according to the loudness of the human voice signal in the main audio stream, or a high-pass filter can be applied to reduce low-frequency environmental noise, avoiding frequency band conflicts between the background audio stream and the human voice signal in the main audio stream, and ensuring that the clarity of the main audio stream is not disturbed.

[0080] The final fused target audio stream achieves a "distinct hierarchy" listening effect: it highlights the core human voice information (such as the explanation content and interactive voice) while retaining the necessary ambient sounds (such as the background sound of the scene), avoiding the problems caused by human voices being masked by ambient sounds or the lack of background sounds in traditional single-channel audio acquisition.

[0081] Subsequently, based on the precise alignment and fusion of the acquisition time, the spatiotemporal consistency of audio and video content is ensured, allowing viewers to have an immersive auditory experience with synchronized sound and clear layers during playback, adapting to the audio needs of various scenarios such as guided tours and interactive shooting.

[0082] In some implementations, the camera device is in the form of glasses and has a microphone for receiving audio streams.

[0083] The glasses are worn in a relatively fixed position on the head of the broadcaster or tour guide. This wearing method allows the shooting equipment to follow the natural rotation of the wearer's head, avoiding the image shake caused by arm tremors, breathing fluctuations, or unstable operation when shooting handheld. At the same time, the center of gravity distribution of the glasses is more ergonomic, reducing equipment displacement during movement, thus obtaining a more stable image effect when capturing video streams, which is especially suitable for achieving smooth, shake-free video recording in dynamic scenarios such as walking and tour guiding.

[0084] In addition, the distance and angle between the microphone and the sound source (such as the wearer's mouth or the surrounding environment) remain constant, making it less likely for the audio signal to fluctuate due to hand shaking or position shift. At the same time, the glasses-shaped microphone can be placed close to the wearer's ear or bridge of the nose, effectively reducing environmental wind noise and signal attenuation caused by long-distance transmission, thereby obtaining a clear, coherent and stable audio stream, which is especially suitable for scenarios that require stable sound recording, such as sports shooting or long-term guided tours.

[0085] Of course, other types of wearable shooting devices, as long as they can keep the shooting device and the wearer relatively still during the shooting process and do not require the photographer to hold them, can be used as the shooting device for this solution. The wearing position can be the wearer's neck, head, or fixed to the wearer's clothing; this solution does not impose any restrictions on this.

[0086] Figure 6 This is a flowchart of another data processing method according to an embodiment of the present disclosure. For example... Figure 6 As shown, this disclosure provides another data processing method, M600, applied to a second device. By parsing the video stream from the first device, especially the orientation information contained in the video stream, it provides the shooting orientation required for the orientation icon to indicate the direction. This allows the second device to drive the orientation icon to rotate synchronously in real time according to the parsed shooting orientation when playing the video stream, accurately mapping the geographical location changes when the shooting device captures the video. This allows viewers to intuitively perceive the shooting angle orientation (such as due north, southeast, etc.) through the orientation icon while watching the video, achieving seamless integration of video content and orientation indication, and greatly improving the orientation recognition efficiency and viewing experience in scenarios such as video navigation.

[0087] The data processing method M600 includes: step S610, receiving a video stream from a first device, the video stream containing orientation information, the orientation information being determined by the first device based on pose sensing data of the shooting device when acquiring the video stream, the orientation information recording the shooting orientation of the shooting device when acquiring each video frame of the video stream; and step S620, displaying an orientation icon representing the shooting orientation when playing the video stream.

[0088] All the technical means involved in this method can be referred to in the previous text, and will not be repeated here.

[0089] In some implementations, displaying an orientation icon representing the shooting orientation includes: when playing a video stream, controlling the rotation component of the orientation icon to display a rotation trajectory about the shooting orientation according to the acquisition sequence of each video frame in the video stream, wherein the trajectory points of the rotation trajectory are used to indicate the shooting orientation corresponding to the currently playing video frame.

[0090] By controlling the rotation component of the orientation icon to display its rotation trajectory according to the acquisition sequence of each video frame in the video stream, frame-level synchronous linkage between the orientation icon and the video screen is achieved. This makes the rotation process of the icon present a smooth and continuous dynamic effect, avoiding the abruptness of traditional static orientation indicators. At the same time, the trajectory points accurately correspond to the shooting orientation of each video frame. Viewers can intuitively perceive the rotation trend of the shooting angle (such as clockwise rotation, counterclockwise deflection, etc.) through the continuous changes in the trajectory. This not only enhances the dynamic continuity of the orientation indicator, but also allows viewers to more clearly understand the camera movement logic of the shooting equipment when watching the video, and obtain a smooth and natural visual experience.

[0091] In some implementations, the rotating component is either a dial or a pointer. The key difference lies in the reference frame for dynamic indication: when the rotating component is a dial, the pointer remains fixed, and the dial rotates synchronously according to the azimuth angle, using the rotation of the scale to reflect the current shooting orientation; when the rotating component is a pointer, the dial remains stationary, and the second device needs to calculate the true north offset between the azimuth angle and the geographic North Pole to drive the pointer to rotate on the dial to indicate direction.

[0092] In some implementations, the orientation icon includes an orientation display control and a rotation direction prompt control, wherein the orientation display control has a rotation component and the rotation direction prompt control is used to display rotation direction information of the shooting device toward the shooting orientation.

[0093] A rotation direction indicator control is essentially a dynamic visual guidance component used to intuitively demonstrate the current rotation trend or target orientation of the shooting equipment. The rotation direction indicator control focuses on solving the dynamic process problem of "how the shooting equipment rotates" or "how the image is changing." This control can be presented in a clear text format, effectively reducing the cognitive load on the viewer and helping them quickly understand the logic behind the changes in the orientation of the display control and the camera movement logic of the video stream.

[0094] Figure 7 This is a schematic diagram of a display interface according to an embodiment of the present disclosure. For example... Figure 7 As shown, an electronic device (i.e., a second device) is displayed with a video stream playback interface on its screen, and an orientation icon, including a rotation direction prompt control and an orientation display control, is overlaid in an area of ​​the playback interface (e.g., the lower right corner).

[0095] The rotation direction prompt control provides a rotation direction prompt, which is to explicitly indicate in text form that "the shooting direction has changed from east to west". Of course, the specific rotation angle and the current azimuth angle can also be specified. The figure is only one possible example.

[0096] The orientation display control adopts a classic compass design, with the rotating component serving as the pointer. During video playback, the external orientation dial remains fixed and stationary, acting as a reference for orientation, while the internal pointer rotates in real time according to the acquisition sequence of video frames, accurately pointing to the orientation angle of the current shooting device relative to the geographic North Pole. Through a combination of static and dynamic visual presentation, viewers can intuitively and clearly perceive the changes in orientation of the shooting perspective while watching the video.

[0097] Figure 8 This is a schematic diagram of another display interface according to an embodiment of the present disclosure. Figure 8 This shows another possible playback interface, in which other parts are similar to... Figure 7The only component is the rotating component, which rotates in real time with the video stream to dynamically display the changes in the shooting device's orientation. The pointer, on the other hand, remains fixed as a reference. The rotation trajectory of the dial visually presents the continuous change in the shooting orientation, allowing viewers to perceive the dynamic evolution of the shooting orientation while watching the video.

[0098] Figure 9 This is a schematic block diagram of the structure of a data processing apparatus according to embodiments of the present disclosure. For example... Figure 9 As shown, this disclosure proposes a data processing device 900, including: a sensor data determination module 910, used to acquire pose sensor data of a shooting device when capturing a video stream; an orientation information calculation module 920, used to determine the orientation information of the shooting device when capturing the video stream based on the pose sensor data, wherein the orientation information records the shooting orientation of the shooting device when capturing each video frame of the video stream; and a push module 930, used to send the video stream containing the orientation information to a second device, wherein the orientation information is used to drive the second device so that the second device displays an orientation icon representing the shooting orientation when playing the video stream.

[0099] The data processing device 900 may further include a receiving module (not shown) for receiving a video stream from a first device, the video stream containing orientation information determined by the first device based on pose sensing data of the shooting device when capturing the video stream, the orientation information recording the shooting orientation of the shooting device when capturing each video frame of the video stream; and a display module (not shown) for displaying an orientation icon representing the shooting orientation when playing the video stream.

[0100] The data processing apparatus 900 disclosed herein may be in the form of computer software, and each module of the data processing apparatus 900 may be in the form of computer software modules.

[0101] The various modules of the data processing apparatus 900 disclosed herein are set up to implement the various steps of the data processing method. Their execution principles and steps can be referred to the above text and will not be repeated here.

[0102] Figure 10 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Figure 10 As shown, this disclosure also provides an electronic device 1000, including: a processor 1200 and a memory 1300, the memory 1300 storing execution instructions; the processor 1200 executes the execution instructions stored in the memory 1300, causing the processor 1200 to execute a data processing method.

[0103] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0104] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0105] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. The readable storage medium can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of readable storage media include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0106] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0107] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0108] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, electronic devices, readable storage media, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In the description of this specification, the references to the terms "an embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0114] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A data processing method, characterized by, Applied to the first device, including: Acquire pose sensing data of the shooting device when capturing video streams; Based on the pose sensing data, the orientation information of the shooting device when acquiring the video stream is determined, and the orientation information records the shooting orientation of the shooting device when acquiring each video frame of the video stream; and A video stream containing the orientation information is sent to a second device, wherein the orientation information is used to drive the second device to display an orientation icon representing the shooting orientation when the video stream is played.

2. The data processing method according to claim 1, characterized in that, Acquire pose sensing data of the shooting device during video stream acquisition, including: Using the magnetometer of the shooting device, magnetometer data is determined when the shooting device acquires the video stream, and the magnetometer data characterizes the three-dimensional magnetic field strength of the shooting device; and Using the sensing unit of the shooting device, the angular velocity and acceleration of the shooting device when acquiring the video stream are determined, and the magnetometer data, the angular velocity and the acceleration are the pose sensing data.

3. The data processing method of claim 1, wherein, Based on the pose sensing data, the orientation information of the shooting device when acquiring the video stream is determined, including: Based on the magnetometer data and acceleration from the pose sensing data, the azimuth angle of the capturing device relative to the geographic North Pole is determined when acquiring the video stream; and Based on the angular velocity in the pose sensing data, the rotation direction of the shooting device when acquiring the video stream is determined, wherein the orientation information includes at least one of the azimuth angle and the rotation direction.

4. The data processing method of claim 1, wherein, Also includes: Based on the first audio stream captured by the shooting device and the second audio stream captured by the audio unit of the first device, a target audio stream suitable for the video stream is determined.

5. The data processing method according to claim 4, characterized in that, Determining the target audio stream suitable for the video stream includes: Analyze the audio content of the first audio stream and the second audio stream, and use the audio stream with human voice signal as the main audio stream and the audio stream without human voice signal as the background audio stream. The main audio stream is boosted, and the background audio stream is attenuated; and According to the acquisition time, the main audio stream after gain enhancement and the background audio stream after attenuation are fused together to form the target audio stream.

6. A data processing method, characterized by, Applied to a second device, including: Receive a video stream from a first device, the video stream containing orientation information determined by the first device based on pose sensing data of a camera device when acquiring the video stream, the orientation information recording the shooting orientation of the camera device when acquiring each video frame of the video stream; and When the video stream is played, an orientation icon representing the shooting orientation is displayed.

7. The data processing method according to claim 6, characterized in that, Display orientation icons representing the shooting orientation, including: When playing the video stream, the rotation component of the orientation icon is controlled to display a rotation trajectory about the shooting orientation according to the acquisition sequence of each video frame in the video stream. The trajectory points of the rotation trajectory are used to indicate the shooting orientation corresponding to the currently playing video frame.

8. The data processing method according to claim 7, characterized in that, The orientation icon includes an orientation display control and a rotation direction prompt control, wherein the rotation direction prompt control is used to display the rotation direction information of the shooting device from the shooting orientation.

9. An electronic device, comprising: include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the data processing method according to any one of claims 1 to 8.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 8.

Images