A multimedia information display method, device and electronic equipment

Abstract

The present disclosure provides a display method and device of multimedia information and electronic equipment, relates to the technical field of display, and solves the problem that a VR device mistakenly considers that a user rotates the VR device. The method comprises the following steps: acquiring a steering wheel rotation angle of a vehicle at a current time and a current image collected by each image collection device; in the case that the steering wheel rotation angle is greater than a rotation angle threshold, at least one actual feature point is extracted according to the current image; a first conversion matrix between a camera coordinate system corresponding to the image collection device and a vehicle coordinate system is determined according to an image coordinate of each actual feature point in the current image and a space coordinate of the actual feature point in the vehicle coordinate system; a theoretical state matrix without a deflection angle is determined according to an actual state matrix and a rotation matrix of the VR device in the device coordinate system at the current time; and the VR device is controlled to display multimedia data according to the theoretical state matrix at a next time.

Description

Technical Field

[0001] This disclosure relates to the field of display technology, and in particular to a method, apparatus and electronic device for displaying multimedia information. Background Technology

[0002] In some scenarios, when users are watching multimedia information while wearing virtual reality (VR) devices, the VR device may mistakenly believe that the user has rotated the device even though the user has not. For example, if the VR device is a VR headset and the user is wearing it while watching multimedia information on a vehicle, and the vehicle makes a turn, the VR headset may mistakenly believe that the user has rotated the headset due to the change in the vehicle's orientation. In this case, the multimedia information played by the VR headset will also rotate accordingly, resulting in a poor user experience. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and electronic device for displaying multimedia information, which solves the problem in related technologies where VR headsets mistakenly believe that the user is rotating the VR headset.

[0004] To achieve the above objectives, the present disclosure adopts the following technical solution:

[0005] In a first aspect, this disclosure provides a method for displaying multimedia information, applied to a vehicle equipped with a VR device. The VR device is equipped with at least one image acquisition device, comprising: acquiring the steering wheel angle of the vehicle at the current moment and the current image acquired by each image acquisition device; when the steering wheel angle is greater than a turning angle threshold, extracting at least one actual feature point based on the current image; determining a first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature point in the vehicle coordinate system; determining a rotation matrix around the Z-axis in the device coordinate system corresponding to the VR device based on the deflection angle of the vehicle's rotation around the Z-axis in the vehicle coordinate system, a pre-configured second transformation matrix, and the first transformation matrix; wherein the second transformation matrix is ​​used to represent the transformation relationship between the coordinates in the camera coordinate system and the coordinates in the device coordinate system; determining a theoretical state matrix after removing the deflection angle based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment; and controlling the VR device to display multimedia data according to the theoretical state matrix at the next moment.

[0006] In some feasible examples, a first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device is determined based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature point in the vehicle coordinate system. This includes: determining a theoretical feature point that matches each actual feature point from a pre-configured set of coordinates; determining the spatial coordinates of each theoretical feature point in the vehicle coordinate system based on a pre-configured spatial relationship and the theoretical feature point that matches each actual feature point; and determining the first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device based on the image coordinates and the spatial coordinates.

[0007] In some feasible examples, before acquiring the steering wheel angle of the vehicle at the current moment and the image information acquired by each image acquisition device, the multimedia information display method provided in this disclosure further includes: acquiring theoretical image information of the vehicle's cabin at different angles acquired by each image acquisition device; extracting at least one theoretical feature point from the theoretical image information; and determining a pre-configured set of coordinates based on at least one theoretical feature point.

[0008] In some implementable examples, the multimedia information display method provided in this disclosure further includes: acquiring a three-dimensional reconstructed model of the cabin of a vehicle; establishing a vehicle coordinate system in the three-dimensional reconstructed model; determining the spatial coordinates of each theoretical feature point in the vehicle coordinate system based on the vehicle coordinate system and at least one theoretical feature point; and determining a pre-configured spatial relationship based on the spatial coordinates of each theoretical feature point in the vehicle coordinate system.

[0009] In some feasible examples, the theoretical state matrix after removing the deflection angle is determined based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment, including: determining the inverse matrix of the rotation matrix based on the rotation matrix; and determining the theoretical state matrix after removing the deflection angle based on the actual state matrix and inverse matrix of the VR device in the device coordinate system at the current moment.

[0010] In some feasible examples, the theoretical state matrix after removing the deflection angle is determined based on the actual state matrix and inverse matrix of the VR device in the device coordinate system at the current moment, including: determining the theoretical state matrix after removing the deflection angle based on the product of the actual state matrix and inverse matrix of the VR device in the device coordinate system at the current moment.

[0011] Secondly, this disclosure provides a multimedia information display device applied to a vehicle equipped with a VR device. The VR device is equipped with at least one image acquisition device, including: an acquisition unit for acquiring the steering wheel angle of the vehicle at the current moment and the current image acquired by each image acquisition device; a processing unit for extracting at least one actual feature point based on the current image acquired by the acquisition unit when the steering wheel angle acquired by the acquisition unit is greater than a turning angle threshold; and the processing unit is further configured to determine the camera coordinates corresponding to the image acquisition device based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature point in the vehicle coordinate system. The processing unit is further configured to: establish a first transformation matrix between the camera coordinate system and the vehicle coordinate system; determine a rotation matrix in the device coordinate system corresponding to the VR device based on the deflection angle of the vehicle's rotation around the Z-axis in the vehicle coordinate system, a pre-configured second transformation matrix, and the first transformation matrix; wherein the second transformation matrix represents the transformation relationship between coordinates in the camera coordinate system and coordinates in the device coordinate system; determine a theoretical state matrix after removing the deflection angle based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment; and control the VR device to display multimedia data according to the theoretical state matrix at the next moment.

[0012] In some feasible examples, the processing unit is specifically used to determine the theoretical feature point that matches each actual feature point from a pre-configured set of coordinates; the processing unit is specifically used to determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system based on the pre-configured spatial relationship and the theoretical feature point that matches each actual feature point; the processing unit is specifically used to determine the first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device based on the image coordinates and the spatial coordinates.

[0013] In some feasible examples, the acquisition unit is further configured to acquire theoretical image information of the cabin of the vehicle at different angles acquired by each image acquisition device; the processing unit is further configured to extract at least one theoretical feature point from the theoretical image information acquired by the acquisition unit; and the processing unit is further configured to determine a pre-configured set of coordinates based on at least one theoretical feature point.

[0014] In some feasible examples, the acquisition unit is further configured to acquire a three-dimensional reconstructed model of the vehicle's cabin; the processing unit is further configured to establish a vehicle coordinate system in the three-dimensional reconstructed model acquired by the acquisition unit; the processing unit is further configured to determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system based on the vehicle coordinate system and at least one theoretical feature point; the processing unit is further configured to determine a pre-configured spatial relationship based on the spatial coordinates of each theoretical feature point in the vehicle coordinate system.

[0015] In some feasible examples, the processing unit is specifically used to determine the inverse matrix of the rotation matrix based on the rotation matrix; the processing unit is specifically used to determine the theoretical state matrix after removing the deflection angle based on the actual state matrix and inverse matrix of the VR device in the device coordinate system at the current moment.

[0016] In some feasible examples, the processing unit is specifically used to determine the theoretical state matrix after removing the deflection angle, based on the product of the actual state matrix and the inverse matrix of the VR device in the device coordinate system at the current moment.

[0017] Thirdly, this disclosure provides an electronic device, including: a communication interface, a processor, a memory, and a bus; the memory stores computer-executed instructions, and the processor is connected to the memory via the bus. When the electronic device is running, the processor executes the computer-executed instructions stored in the memory to cause the electronic device to perform the multimedia information display method provided in the first aspect above.

[0018] Fourthly, this disclosure provides a computer-readable storage medium including instructions. When the instructions are executed on a computer, the computer performs the method for displaying multimedia information as described in the first aspect above.

[0019] Fifthly, this disclosure provides a computer program product that, when run on a computer, causes the computer to perform a multimedia information display method as described in the design of the first aspect.

[0020] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be packaged together with the processor of the multimedia information display device, or it may be packaged separately from the processor of the multimedia information display device; this disclosure does not impose any limitations on this.

[0021] The descriptions of the second, third, fourth, and fifth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, fourth, and fifth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0022] In this disclosure, the name of the multimedia information display device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of this disclosure, it falls within the scope of the claims of this disclosure and its equivalents.

[0023] These or other aspects of this disclosure will become more readily apparent in the following description.

[0024] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0025] By determining the relationship between the steering wheel angle of the vehicle at the current moment and a turning angle threshold, it can be determined whether the vehicle is turning. If the steering wheel angle is greater than the turning angle threshold, it indicates that the vehicle is turning, which may cause the VR device to mistakenly perceive that the user is rotating the VR device when it is stationary. Then, by extracting feature points from the current image captured by the image acquisition device on the VR device, at least one actual feature point can be determined. Based on the image coordinates of each actual feature point in the current image and its spatial coordinates in the vehicle coordinate system, a first transformation matrix between the camera coordinate system and the vehicle coordinate system is determined. Since the car can be considered to be rotating around the z-axis in the vehicle coordinate system with a certain deflection angle during a turn, and the VR device is stationary, the car's rotation may be mistakenly perceived as the user's head turning. Therefore, it is necessary to determine the rotation matrix that the VR device mistakenly perceives as rotation. For example, based on the deflection angle of the car's rotation around the Z-axis in the vehicle coordinate system, the pre-configured second transformation matrix, and the first transformation matrix, the rotation matrix of the VR device around the Z-axis in the device coordinate system is determined. Furthermore, based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment, a theoretical state matrix is ​​determined to remove the deflection angle. This eliminates the rotation of the car, so even if the car is turning, the multimedia data displayed by the VR device will not rotate, ensuring the user experience and solving the problem in related technologies where the VR device mistakenly believes that the user is rotating the VR device. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is one of the flowcharts illustrating a method for displaying multimedia information according to an embodiment of this disclosure;

[0029] Figure 2 A schematic diagram of the camera coordinate system for a multimedia information display method provided in an embodiment of this disclosure;

[0030] Figure 3A second schematic flowchart illustrating a method for displaying multimedia information provided in this embodiment of the present disclosure;

[0031] Figure 4 A third schematic flowchart illustrating a method for displaying multimedia information provided in this embodiment of the present disclosure;

[0032] Figure 5 This is one of the structural schematic diagrams of a control device for an automotive air conditioner provided in an embodiment of this disclosure;

[0033] Figure 6 This is a second schematic diagram of the structure of a control device for an automotive air conditioner provided in an embodiment of this disclosure;

[0034] Figure 7 This is a schematic diagram of the structure of a computer program product for a multimedia information display method provided in an embodiment of this disclosure. Detailed Implementation

[0035] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0036] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0037] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0038] In the multimedia information display method provided in this disclosure, VR device refers to virtual reality device. It was also previously referred to as VR glasses or VR helmet. A VR device is a head-mounted display that blocks a person's vision and hearing from the outside world, guiding the user to feel as if they are in a virtual environment.

[0039] In the multimedia information display method provided in this disclosure, VR glasses refer to VR devices, specifically virtual reality head-mounted display devices. Since the concept of a head-mounted display didn't exist initially, unprofessional names such as VR glasses, VR goggles, and VR helmets arose based on their appearance. A VR head-mounted display uses a head-mounted display device to block out a person's vision and hearing from the outside world, guiding the user to experience a feeling of being in a virtual environment. Its display principle involves the left and right eye screens displaying images for the left and right eyes respectively; after the human eye receives this differentiated information, a sense of depth is generated in the brain.

[0040] In the multimedia information display method provided in this disclosure, OpenCV, or "Open Source Computer Vision Library," is a cross-platform, open-source, mid-to-high-level "Application Programming Interface" (API) development library consisting of more than 500 C functions, established by Intel in 1999.

[0041] It should be noted that when a vehicle executes the multimedia information display method provided in this embodiment, the executing entity is the vehicle's processor. For example, when the vehicle is a car, the executing entity for the multimedia information display method provided in this embodiment is the car's processor.

[0042] For example, taking a car as an example, the method for displaying multimedia information provided in this disclosure will be described. Figure 1 The method for displaying multimedia information includes the following steps S11-S16:

[0043] S11. Obtain the steering wheel angle of the vehicle at the current moment and the current image captured by each image acquisition device.

[0044] In some examples, the car cabin is equipped with an interface for connecting VR devices. Users can connect their VR devices to the car through this interface after entering the vehicle. Exemplary VR devices include either a VR headset or VR glasses.

[0045] In some examples, the multimedia information display method provided in this disclosure, by fixing the image acquisition device on the side of the VR device away from the face, can determine the spatial position of the VR device inside the car cabin based on the image acquired by the image acquisition device. Exemplarily, the image acquisition device includes a camera.

[0046] S12. When the steering wheel angle is greater than the angle threshold, extract at least one actual feature point based on the current image.

[0047] In some examples, the VR device is fixed to the user's head, so the turning angle threshold is equal to the steering wheel angle that the VR device determines when the user's head moves while the VR device is stationary. For example, the turning angle threshold is 5°.

[0048] Specifically, the actual feature points can be the four corners of the car's windshield, the four corners of the car's rearview mirror, or any point on the car's steering wheel.

[0049] Specifically, the more actual feature points extracted, the more accurately the car can determine the spatial location of the VR device within the car's cabin. For example, based on the current image, identifying at least six actual feature points ensures that the car can accurately determine the spatial location of the VR device within the car's cabin.

[0050] In some examples, the car returns to executing S11 if the steering wheel angle is less than or equal to the angle threshold.

[0051] S13. Based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature point in the vehicle coordinate system, determine the first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device.

[0052] In some examples, the origin of the camera coordinate system (also called the observation coordinate system) is the optical center of the image acquisition device. The x-axis and y-axis are parallel to the X and Y axes of the image, and the z-axis is the optical axis of the image acquisition device, which is perpendicular to the image plane. The intersection of the optical axis and the image plane is the origin of the image coordinate system, which is a two-dimensional rectangular coordinate system.

[0053] In some examples, to determine the spatial position of a VR device inside a car cabin, feature extraction is needed from the current image captured by the VR device's image acquisition device to identify at least one actual feature point. Then, by analyzing the actual feature points in each current image, the image coordinates (also called pixel coordinates) of each actual feature point can be obtained. To determine the pixel coordinates corresponding to each actual feature point, a coordinate system needs to be established. This coordinate system could be a camera coordinate system. Figure 2 As shown, point o in the camera coordinate system is the camera's optical center (projection center). The Xc and Yc axes are parallel to the x and y axes of the imaging plane coordinate system, and the Zc axis is the camera's optical axis, perpendicular to the image plane. The intersection of the optical axis and the image plane is the principal point o1 of the image. The rectangular coordinate system formed by point o and the Xc, Yc, and Zc axes is called the camera's coordinate system. oo1 is the camera's focal length. In this way, the car's processor can determine the image coordinates and spatial coordinates of the actual feature points, and determine the first transformation matrix between the camera coordinate system and the vehicle coordinate system.

[0054] S14. Based on the deflection angle of the vehicle's rotation around the Z-axis in the vehicle coordinate system, the pre-configured second transformation matrix, and the first transformation matrix, determine the rotation matrix of the VR device's rotation around the Z-axis in the device coordinate system. The second transformation matrix represents the transformation relationship between coordinates in the camera coordinate system and coordinates in the device coordinate system.

[0055] In some examples, when a car turns, its motion can be represented as a rotation of the car around the z-axis in the car's coordinate system by a deflection angle. For ease of calculation, this deflection angle can be converted into a first deflection amount. Then, based on the product of this first deflection amount and a first transformation matrix, a second deflection amount in the camera's coordinate system is determined. Since the transformation relationship between coordinates in the camera's coordinate system and coordinates in the device's coordinate system can be represented by a second transformation matrix, a third deflection amount in the device's coordinate system is determined based on the product of the second deflection amount and the second transformation matrix. Thus, the car's rotation around the z-axis in the car's coordinate system by a deflection angle can be converted into the rotation angle of the VR device around the z-axis in the device's coordinate system. For ease of description, this rotation angle can be represented as a rotation matrix. Therefore, it can be determined that when the car turns, even if the VR device remains stationary, the VR device will misinterpret the rotation angle around the z-axis in the device's coordinate system. After removing this rotation angle, it can be ensured that the VR device will not misinterpret the car's rotation around the z-axis in the device's coordinate system when turning.

[0056] Specifically, the first rotation matrix includes a first rotation matrix R and a first translation matrix t, and the second transformation matrix includes a second transformation matrix R and a second translation matrix t. The first rotation matrix and the second transformation matrix are different.

[0057] Specifically, the deflection angle is equal to a × steering wheel angle, where a is a constant greater than 0.

[0058] Specifically, the origin o of the device coordinate system is the center point of the VR device, the y-axis is along the direction of the car's front, the x-axis is perpendicular to the y-axis and horizontal to the right, the z-axis is perpendicular to the xoy plane, and the xoy plane is parallel to the ground plane.

[0059] Specifically, the calculation process of the second transformation matrix between the coordinates in the device coordinate system and the coordinates in the camera coordinate system is similar to the calculation process between the coordinates in the camera coordinate system and the coordinates in the vehicle coordinate system, and will not be repeated here.

[0060] S15. Based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment, determine the theoretical state matrix after removing the deflection angle.

[0061] S16. Control the VR device to display multimedia data according to the theoretical state matrix in the next moment.

[0062] As described above, the multimedia information display method provided in this embodiment can determine whether a car is turning by judging the relationship between the steering wheel angle of the car at the current moment and the angle threshold. If the steering wheel angle is greater than the angle threshold, it indicates that the car is turning, which may cause the VR headset to mistakenly believe that the user is rotating when the VR device is stationary. Then, by extracting feature points from the current image captured by the image acquisition device on the VR device, at least one actual feature point can be determined. Based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature point in the vehicle coordinate system, a first transformation matrix between the camera coordinate system and the vehicle coordinate system is determined. Since the car can be seen as rotating around the z-axis in the vehicle coordinate system with a certain deflection angle during turning, and the VR device remains stationary, the car's rotation may mistakenly indicate that the user's head has rotated. Therefore, it is necessary to determine the rotation matrix that the VR device mistakenly believes has occurred. For example, based on the deflection angle of the car's rotation around the Z-axis in the vehicle coordinate system, the pre-configured second transformation matrix, and the first transformation matrix, the rotation matrix of the VR device around the Z-axis in the device coordinate system is determined. Furthermore, based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment, a theoretical state matrix is ​​determined to remove the deflection angle. This eliminates the possibility that the car's rotation might cause the VR device to mistakenly believe that the user's head has rotated. Subsequently, the car controls the VR device to display multimedia data according to the theoretical state matrix at the next moment. Thus, even if the car is turning, the multimedia data displayed by the VR device will not rotate when the VR device remains stationary, ensuring a smooth user experience.

[0063] In some feasible examples, combining Figure 1 ,like Figure 3 As shown, the above S13 can be specifically implemented through the following S130-S132.

[0064] S130. Determine the theoretical feature points that match each actual feature point from the pre-configured set of coordinates.

[0065] Specifically, upon initial operation, the VR device captures images of the car's interior and extracts features from them, identifying at least one theoretical feature point and a corresponding feature descriptor for each point. Based on these theoretical feature points and their corresponding descriptors, a pre-configured set of coordinates can be determined. Subsequent use of the VR device by the user allows for feature matching between the extracted feature descriptors of the actual and theoretical feature points, thus identifying the matching theoretical feature point.

[0066] S131. Based on the pre-configured spatial relationships and the theoretical feature points that match each actual feature point, determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system.

[0067] Specifically, when the VR device is first run, a vehicle coordinate system can be established within the point cloud model of the car's cabin, and each theoretical feature point can be mapped into this vehicle coordinate system. This determines the spatial coordinates corresponding to each theoretical feature point. Based on these spatial coordinates, pre-configured spatial relationships can then be determined. Subsequent times when the user uses the VR device, feature matching can be performed between the extracted actual and theoretical feature points to determine the theoretical feature points that match the actual feature points. Since each theoretical feature point corresponds to a spatial coordinate, the spatial coordinates corresponding to each actual feature point can also be determined.

[0068] S132. Based on the image coordinates and spatial coordinates, determine the first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device.

[0069] In some examples, the car's processor solves for the first transformation matrix between the camera coordinate system and the vehicle coordinate system based on image coordinates, spatial coordinates, and the solvePnP function in OpenCV.

[0070] In some feasible examples, combining Figure 1 ,like Figure 3 As shown, the multimedia information display method provided in this embodiment of the present disclosure further includes: S17-S19.

[0071] S17. Acquire theoretical image information of the interior of the vehicle from different angles by each image acquisition device.

[0072] In some examples, the theoretical image information acquired by the car through the image acquisition device needs to cover all parts of the car's cabin to ensure the accuracy of the VR device's spatial location within the car's cabin, which can be determined by the car's processor.

[0073] Specifically, the theoretical image information acquired by the car through the image acquisition device contains duplicate feature points. It is necessary to remove duplicate feature points and keep only the unique one.

[0074] S18. Extract at least one theoretical feature point from the theoretical image information.

[0075] S19. Determine a pre-configured set of coordinates based on at least one theoretical feature point.

[0076] In some feasible examples, combining Figure 1 ,like Figure 3As shown, the multimedia information display method provided in this embodiment of the present disclosure further includes: S20-S23.

[0077] S20. Obtain a 3D remodel of the vehicle's cabin.

[0078] In some examples, the 3D reconstruction model can be a 3D point cloud model, which can be a 1:1 3D point cloud model of the car.

[0079] S21. Establish a vehicle coordinate system in the 3D reconstruction model. The y-axis of the vehicle coordinate system is along the direction of the car's front, the x-axis is perpendicular to the y-axis and points horizontally to the right, the origin of the vehicle coordinate system is the center of the car, the xoy plane of the vehicle coordinate system is parallel to the ground plane, the z-axis of the vehicle coordinate system is perpendicular to the xoy plane, and o represents the origin of the vehicle coordinate system.

[0080] In some examples, the origin of the vehicle coordinate system can be any point inside the vehicle's cabin. Preferably, the origin of the vehicle coordinate system is the center of the vehicle.

[0081] S22. Based on the vehicle coordinate system and at least one theoretical feature point, determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system.

[0082] S23. Determine the pre-configured spatial relationships based on the spatial coordinates of each theoretical feature point in the vehicle coordinate system.

[0083] In some feasible examples, combining Figure 1 ,like Figure 3 As shown, the above S15 can be implemented by the following S150 and S151.

[0084] S150. Determine the inverse moment of the rotation matrix based on the rotation matrix.

[0085] In some examples, the inverse of the rotation matrix is ​​equal to the transpose of the rotation matrix.

[0086] S151. Based on the actual state matrix and inverse matrix of the VR device in the device coordinate system at the current moment, determine the theoretical state matrix after removing the deflection angle.

[0087] In some examples, the car determines the theoretical state matrix, excluding the deflection angle, based on the difference between the actual state matrix and the inverse matrix of the VR device in the device coordinate system at the current moment. Alternatively, the car determines the theoretical state matrix, excluding the deflection angle, based on the sum of the actual state matrix and the inverse matrix of the VR device in the device coordinate system at the current moment.

[0088] In some feasible examples, combining Figure 3 ,like Figure 4As shown, the above S151 can be specifically implemented through the following S1510.

[0089] S1510. Determine the theoretical state matrix after removing the deflection angle based on the product of the actual state matrix and the inverse matrix of the VR device in the device coordinate system at the current moment.

[0090] The above example illustrates the multimedia information display method provided in this disclosure using a car as an example. In other examples, the device executing the multimedia information display method provided in this disclosure can also be a VR device. In this case, the VR device needs to acquire the steering wheel angle of the car at the current moment and the current image acquired by each image acquisition device. When the steering wheel angle is greater than the angle threshold, the VR device extracts at least one actual feature point based on the current image; the VR device determines a first transformation matrix between the camera coordinate system and the vehicle coordinate system based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature point in the vehicle coordinate system; the VR device determines a rotation matrix of the VR device around the Z-axis in the device coordinate system based on the deflection angle of the car's rotation around the Z-axis in the vehicle coordinate system, a pre-configured second transformation matrix, and the first transformation matrix; wherein, the second transformation matrix is ​​used to represent the transformation relationship between the coordinates in the camera coordinate system and the coordinates in the device coordinate system; the VR device determines a theoretical state matrix after removing the deflection angle based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment; the VR device displays multimedia data according to the theoretical state matrix at the next moment. In this way, the VR device can adjust the theoretical state matrix of the displayed multimedia data in real time according to the car's movement posture, ensuring the user experience.

[0091] Specifically, the process by which the VR device determines the first transformation matrix is ​​similar to that of the car, and will not be elaborated here.

[0092] Specifically, the process by which a VR device determines a pre-configured set of coordinates is similar to that of a car, and will not be elaborated here.

[0093] Specifically, the process by which VR devices determine pre-configured spatial relationships is similar to that of automobiles, and will not be elaborated here.

[0094] Specifically, the process by which a VR device determines the theoretical state matrix after removing the deflection angle is similar to that of a car, and will not be elaborated here.

[0095] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0096] This disclosure embodiment can divide the multimedia information display device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0097] like Figure 5 As shown, a multimedia information display device 10 provided in this embodiment of the present disclosure is applied to a car that can be equipped with a VR device. The VR device is equipped with at least one image acquisition device. The multimedia information display device 10 is used to acquire the steering wheel angle of the vehicle at the current moment and the current image acquired by each image acquisition device; when the steering wheel angle is greater than the angle threshold, at least one actual feature point is extracted based on the current image; based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature point in the vehicle coordinate system, a first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device is determined; based on the deflection angle of the vehicle rotating around the Z-axis in the vehicle coordinate system, the pre-configured second transformation matrix, and the first transformation matrix, a rotation matrix around the Z-axis in the device coordinate system corresponding to the VR device is determined; wherein, the second transformation matrix is ​​used to represent the transformation relationship between the coordinates in the camera coordinate system and the coordinates in the device coordinate system; based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment, a theoretical state matrix after removing the deflection angle is determined; and the VR device is controlled to display multimedia data according to the theoretical state matrix at the next moment. The multimedia information display device 10 includes an acquisition unit 101 and a processing unit 102.

[0098] The acquisition unit 101 is used to acquire the steering wheel angle of the vehicle at the current moment and the current image acquired by each image acquisition device; the processing unit 102 is used to extract at least one actual feature point based on the current image acquired by the acquisition unit 101 when the steering wheel angle acquired by the acquisition unit 101 is greater than the angle threshold; the processing unit 102 is also used to determine a first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature point in the vehicle coordinate system; the processing unit 102 is also used to... Based on the deflection angle of the vehicle's rotation around the Z-axis in the vehicle coordinate system, the pre-configured second transformation matrix, and the first transformation matrix, the processing unit 102 determines the rotation matrix of the VR device's rotation around the Z-axis in the device coordinate system. The second transformation matrix is ​​used to represent the transformation relationship between the coordinates in the camera coordinate system and the coordinates in the device coordinate system. The processing unit 102 is also used to determine the theoretical state matrix after removing the deflection angle based on the actual state matrix and rotation matrix of the VR device in the device coordinate system at the current moment. The processing unit 102 is also used to control the VR device to display multimedia data according to the theoretical state matrix at the next moment.

[0099] In some implementable examples, processing unit 102 is specifically used to determine the theoretical feature point that matches each actual feature point in a pre-configured set of coordinates; processing unit 102 is specifically used to determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system according to the pre-configured spatial relationship and the theoretical feature point that matches each actual feature point; processing unit 102 is specifically used to determine the first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device according to the image coordinates and the spatial coordinates.

[0100] In some feasible examples, the acquisition unit 101 is further configured to acquire theoretical image information of the cabin of the vehicle at different angles acquired by each image acquisition device; the processing unit 102 is further configured to extract at least one theoretical feature point from the theoretical image information acquired by the acquisition unit 101; the processing unit 102 is further configured to determine a pre-configured set of coordinates based on at least one theoretical feature point.

[0101] In some feasible examples, the acquisition unit 101 is further configured to acquire a three-dimensional reconstructed model of the vehicle's cabin; the processing unit 102 is further configured to establish a vehicle coordinate system in the three-dimensional reconstructed model acquired by the acquisition unit 101; the processing unit 102 is further configured to determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system based on the vehicle coordinate system and at least one theoretical feature point; the processing unit 102 is further configured to determine a pre-configured spatial relationship based on the spatial coordinates of each theoretical feature point in the vehicle coordinate system.

[0102] In some implementable examples, processing unit 102 is specifically used to determine the inverse matrix of the rotation matrix based on the rotation matrix; processing unit 102 is specifically used to determine the theoretical state matrix after removing the deflection angle based on the actual state matrix and inverse matrix of the VR device in the device coordinate system at the current moment.

[0103] In some implementable examples, the processing unit 102 is specifically used to determine the theoretical state matrix after removing the deflection angle based on the product of the actual state matrix and the inverse matrix of the VR device in the device coordinate system at the current moment.

[0104] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and their functions will not be repeated here.

[0105] Of course, the multimedia information display device 10 provided in this embodiment includes, but is not limited to, the modules described above. For example, the multimedia information display device 10 may also include a storage unit 103. The storage unit 103 may be used to store the program code of the multimedia information display device 10, and may also be used to store data generated by the multimedia information display device 10 during operation, such as data in a write request.

[0106] Figure 6 This is a schematic diagram of the structure of a multimedia information display device 10 provided in an embodiment of the present disclosure, as shown below. Figure 6 As shown, the multimedia information display device 10 may include at least one processor 51, a memory 52, a communication interface 53, and a communication bus 54.

[0107] The following is combined with Figure 6 The various components of the multimedia information display device 10 will be described in detail below:

[0108] The processor 51 is the control center of the multimedia information display device 10. It can be a single processor or a collective term for multiple processing elements. For example, the processor 51 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this disclosure, such as one or more DSPs, or one or more field-programmable gate arrays (FPGAs).

[0109] In a specific implementation, as one example, processor 51 may include one or more CPUs, for example... Figure 6CPU0 and CPU1 are shown in the diagram. Furthermore, as one embodiment, the multimedia information display device 10 may include multiple processors, such as... Figure 6 The processors 51 and 55 are shown. Each of these processors can be a single-core processor or a multi-core processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0110] The memory 52 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or it may be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 52 may exist independently and be connected to the processor 51 via a communication bus 54. The memory 52 may also be integrated with the processor 51.

[0111] In a specific implementation, memory 52 is used to store data disclosed herein and execute software programs disclosed herein. Processor 51 can perform various functions of the air conditioner by running or executing software programs stored in memory 52 and by calling data stored in memory 52.

[0112] Communication interface 53 uses any transceiver-like device for communicating with other devices or communication networks, such as Radio Access Network (RAN), Wireless Local Area Networks (WLAN), terminals, and the cloud. Communication interface 53 may include acquisition unit 101 to implement acquisition functions.

[0113] The communication bus 54 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0114] As an example, combined Figure 5 The acquisition unit 101 in the multimedia information display device 10 performs the same functions as... Figure 6 The communication interface 53 has the same function, and the processing unit 102 implements the same function. Figure 6 The processor 51 in the memory unit has the same function as the memory unit 103. Figure 6 The memory 52 in it has the same function.

[0115] Another embodiment of this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods shown in the above-described method embodiments.

[0116] In some embodiments, the disclosed method may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0117] Figure 7 A conceptual partial view of a computer program product provided in an embodiment of the present disclosure is shown schematically. The computer program product includes a computer program for executing computer processes on a computing device.

[0118] In one embodiment, a computer program product is provided using a signal bearer medium 410. The signal bearer medium 410 may include one or more program instructions that, when executed by one or more processors, can provide the above-mentioned... Figure 1 The described function or part of the function. Therefore, for example, refer to... Figure 1 In the embodiment shown, one or more features of S11-S16 can be fulfilled by one or more instructions associated with the signal carrying medium 410. Furthermore, Figure 7 The program instructions in the document also describe example instructions.

[0119] In some examples, the signal carrying medium 410 may include a computer-readable medium 411, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, a read-only memory (ROM), or a random access memory (RAM), etc.

[0120] In some implementations, the signal carrying medium 410 may include a computer recordable medium 412, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on.

[0121] In some implementations, the signal carrying medium 410 may include a communication medium 413, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.).

[0122] The signal-bearing medium 410 can be transmitted by a wireless communication medium 413 (e.g., a wireless communication medium conforming to the IEEE 802.41 standard or other transmission protocols). One or more program instructions can be, for example, computer-executable instructions or logical implementation instructions.

[0123] In some examples, such as targeting Figure 1 The described data writing device can be configured to provide various operations, functions, or actions in response to one or more program instructions in a computer-readable medium 411, a computer-recordable medium 412, and / or a communication medium 413.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0125] In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0126] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0129] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for displaying multimedia information, characterized in that, Applicable to vehicles equipped with VR devices, wherein the VR devices are equipped with at least one image acquisition device, including: The steering wheel angle of the vehicle at the current moment and the current image captured by each of the image acquisition devices are obtained; If the steering wheel angle is greater than the angle threshold, at least one actual feature point is extracted based on the current image; Based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature points in the vehicle coordinate system, a first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device is determined. Based on the deflection angle of the vehicle's rotation around the Z-axis in the vehicle coordinate system, the pre-configured second transformation matrix, and the first transformation matrix, the rotation matrix of the VR device's rotation around the Z-axis in the device coordinate system is determined; wherein, the second transformation matrix is ​​used to represent the transformation relationship between the coordinates in the camera coordinate system and the coordinates in the device coordinate system; Based on the actual state matrix of the VR device in the device coordinate system at the current moment and the rotation matrix, determine the theoretical state matrix after removing the deflection angle; The VR device is controlled to display multimedia data according to the theoretical state matrix in the next moment.

2. The method for displaying multimedia information according to claim 1, characterized in that, The step of determining the first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device, based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature points in the vehicle coordinate system, includes: Determine theoretical feature points that match each actual feature point from a pre-configured set of coordinates; Based on the pre-configured spatial relationships and the theoretical feature points that match each actual feature point, the spatial coordinates of each theoretical feature point in the vehicle coordinate system are determined. Based on the image coordinates and the spatial coordinates, a first transformation matrix is ​​determined between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device.

3. The method for displaying multimedia information according to claim 2, characterized in that, Before acquiring the steering wheel angle of the vehicle at the current moment and the image information acquired by each of the image acquisition devices, the method further includes: Theoretical image information of the interior of the vehicle from different angles is acquired by each of the image acquisition devices; Extract at least one theoretical feature point from the theoretical image information; Based on the at least one theoretical feature point, a pre-configured set of coordinates is determined.

4. The method for displaying multimedia information according to claim 3, characterized in that, The method further includes: Obtain a three-dimensional reconstructed model of the interior of the vehicle's cabin; Establish the vehicle coordinate system in the three-dimensional reconstruction model; Based on the vehicle coordinate system and the at least one theoretical feature point, determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system; The pre-configured spatial relationship is determined based on the spatial coordinates of each theoretical feature point in the vehicle coordinate system.

5. The method for displaying multimedia information according to claim 1, characterized in that, The step of determining the theoretical state matrix after removing the deflection angle based on the actual state matrix of the VR device in the device coordinate system at the current moment and the rotation matrix includes: Based on the rotation matrix, determine the inverse matrix of the rotation matrix; Based on the actual state matrix of the VR device in the device coordinate system at the current moment and the inverse matrix, determine the theoretical state matrix after removing the deflection angle.

6. The method for displaying multimedia information according to claim 5, characterized in that, The step of determining the theoretical state matrix after removing the deflection angle based on the actual state matrix of the VR device in the device coordinate system at the current moment and the inverse matrix includes: The theoretical state matrix after removing the deflection angle is determined based on the product of the actual state matrix of the VR device in the device coordinate system at the current moment and the inverse matrix.

7. A multimedia information display device, characterized in that, Applicable to vehicles equipped with VR devices, wherein the VR devices are equipped with at least one image acquisition device, including: The acquisition unit is used to acquire the steering wheel angle of the vehicle at the current moment and the current image acquired by each of the image acquisition devices; The processing unit is configured to extract at least one actual feature point based on the current image obtained by the acquisition unit when the steering wheel angle obtained by the acquisition unit is greater than the angle threshold. The processing unit is further configured to determine a first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device based on the image coordinates of each actual feature point in the current image and the spatial coordinates of the actual feature points in the vehicle coordinate system. The processing unit is further configured to determine the rotation matrix of the VR device in the device coordinate system around the Z-axis based on the deflection angle of the vehicle's rotation around the Z-axis in the vehicle coordinate system, the pre-configured second transformation matrix, and the first transformation matrix; wherein, the second transformation matrix is ​​used to represent the transformation relationship between the coordinates in the camera coordinate system and the coordinates in the device coordinate system; The processing unit is further configured to determine a theoretical state matrix after removing the deflection angle based on the actual state matrix of the VR device in the device coordinate system at the current moment and the rotation matrix. The processing unit is also used to control the VR device to display multimedia data according to the theoretical state matrix in the next moment.

8. The multimedia information display device according to claim 7, characterized in that, The processing unit is specifically used to determine theoretical feature points that match each actual feature point from a pre-configured set of coordinates; The processing unit is specifically used to determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system based on the pre-configured spatial relationship and the theoretical feature points that match each actual feature point. The processing unit is specifically used to determine a first transformation matrix between the camera coordinate system and the vehicle coordinate system corresponding to the image acquisition device, based on the image coordinates and the spatial coordinates.

9. The multimedia information display device according to claim 8, characterized in that, The acquisition unit is also used to acquire theoretical image information of the cabin of the vehicle at different angles acquired by each of the image acquisition devices; The processing unit is further configured to extract at least one theoretical feature point from the theoretical image information acquired by the acquisition unit; The processing unit is further configured to determine a pre-configured set of coordinates based on the at least one theoretical feature point.

10. The multimedia information display device according to claim 9, characterized in that, The acquisition unit is also used to acquire a three-dimensional reconstruction model of the cabin of the vehicle; The processing unit is also used to establish the vehicle coordinate system in the three-dimensional reconstruction model acquired by the acquisition unit; The processing unit is further configured to determine the spatial coordinates of each theoretical feature point in the vehicle coordinate system based on the vehicle coordinate system and the at least one theoretical feature point; The processing unit is further configured to determine the pre-configured spatial relationship based on the spatial coordinates of each theoretical feature point in the vehicle coordinate system.

11. The multimedia information display device according to claim 7, characterized in that, The processing unit is specifically used to determine the inverse matrix of the rotation matrix based on the rotation matrix; The processing unit is specifically used to determine the theoretical state matrix after removing the deflection angle based on the actual state matrix of the VR device in the device coordinate system at the current moment and the inverse matrix.

12. The multimedia information display device according to claim 11, characterized in that, The processing unit is specifically used to determine the theoretical state matrix after removing the deflection angle based on the product of the actual state matrix of the VR device in the device coordinate system at the current moment and the inverse matrix.

13. An electronic device, characterized in that, include: Communication interface, processor, memory, bus; The memory is used to store computer execution instructions, and the processor is connected to the memory via the bus; When the electronic device is running, the processor executes computer execution instructions stored in the memory to cause the electronic device to perform the multimedia information display method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, Including instructions that, when executed on a computer, cause the computer to perform a method for displaying multimedia information as described in any one of claims 1-6.

Images