Method and device for generating augmented reality graphic content representative of a vehicle collision test

Abstract

The present invention relates to a method and device for generating augmented reality graphic content representing a vehicle crash test. To this end, initial data representing a three-dimensional video sequence of a vehicle crash test involving a crash dummy are received (31). A digital twin is selected (32) from a plurality of digital twins, each digital twin of the plurality of digital twins being representative of a three-dimensional model of a dummy representing a person with a specific morphology.The augmented reality graphic content is generated (34) from the first and second data received (33) and corresponds to a three-dimensional model of the mannequin associated with the selected digital twin by replacing the mannequin in the three-dimensional video sequence with the three-dimensional model of the mannequin associated with the selected digital twin. Figure for the abstract: Figure 3.

Description

Title of the invention: Method and device for generating augmented reality graphic content representing a vehicle collision test technical field

[0001] The present invention relates to methods and devices for generating augmented reality graphic content representing a vehicle crash test, particularly, but not exclusively, of a motor vehicle. The present invention also relates to methods and devices for simulating vehicle crash tests in augmented reality. Technological background

[0002] To improve vehicle safety and verify their behavior in specific situations, it is common practice to conduct a series of tests before a new vehicle is placed on the market. Among these tests, crash tests are performed to assess vehicle behavior in the event of a collision, for example, a frontal collision, a side collision, or a collision with a pole. Some of these tests are mandatory for authorizing the registration of vehicles in a given territory. For example, the European Advanced Motor Safety Committee has defined a test, known as the offset frontal test, as a basis for legislation imposed on all vehicles registered in the European Union.

[0003] Different methods are used to analyze the behavior of vehicles during a collision, namely the experimental or empirical method and the numerical or simulation method.

[0004] The experimental method consists of projecting the vehicle into an obstacle at a given speed to analyze the deformation of the vehicle's various components and to measure the biomechanical stresses experienced by one or more crash test dummies representing one or more occupants (for example, the driver and front passenger) during the collision. The experimental method has the advantage of providing realistic data on all the stresses experienced by the vehicle and the dummies during the test. However, such a method is costly, particularly when it comes to reproducing different real-world vehicle situations.

[0005] The numerical method has the advantage of being less expensive than the experimental method, but the data obtained from the simulation are subject to error or inaccuracy. Summary of the present invention

[0006] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0007] An object of the present invention is, for example, to improve the numerical simulation of a collision test.

[0008] Another object of the present invention is, for example, to improve the consideration of different occupant morphologies in a vehicle collision test.

[0009] According to a first aspect, the present invention relates to a method for generating augmented reality graphic content representing a vehicle collision test, the method being implemented by at least one processor and comprising the following steps: - receipt of initial representative data from a three-dimensional video sequence of a vehicle crash test carrying a dummy, the initial data including geometric information associated with the dummy and a set of elements forming the vehicle; - selection of a digital twin, called the selected digital twin, from a plurality of digital twins, each digital twin of the plurality of digital twins being representative of a three-dimensional model of a mannequin representative of a person having a determined morphology; - reception of second data representing a three-dimensional model of the mannequin associated with the selected digital twin; - generation of augmented reality graphic content from the first and second data by replacing the mannequin in the three-dimensional video sequence with the three-dimensional modeling of the mannequin associated with the selected digital twin.

[0010] Such a method makes it possible to simulate, in augmented reality, a collision test with a multitude of dummies of different morphologies. For each digital test, a digital twin representing a dummy of a particular morphology is selected from a list of different digital twins. The modeling data of the dummy associated with the selected digital twin is used to replace the vehicle dummy that has undergone an experimental collision test in the 3D video of this collision test, thereby generating augmented reality graphic content of this collision test with the dummy corresponding to the selected digital twin.

[0011] This allows observation of the mannequin's behavior during the collision test in augmented reality, thereby improving the sense of immersion during the collision test and enabling a better understanding of the impact of the collision on the mannequin. Such a method makes it possible to simulate the collision test with different body shapes. mannequins obtained from different digital twins of these mannequins at a lower cost with high realism.

[0012] According to one variant, the method further includes a step of displaying the graphic content in augmented reality on a display device.

[0013] According to yet another variant, the first data are generated by composing image data received from a plurality of cameras configured for the acquisition of images of the collision test from a plurality of viewpoints.

[0014] According to another variant, the plurality of digital twins is generated from statistical morphology data of a set of vehicle drivers.

[0015] According to a further variant, each digital twin of the plurality of digital twins is recorded in a blockchain.

[0016] According to an additional variant, the geometric information and the second data include representative finite element data.

[0017] According to a second aspect, the present invention relates to a device for generating augmented reality graphic content representative of a vehicle collision test, the device comprising a memory associated with a processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0018] According to a third aspect, the present invention relates to a data communication system comprising a device as described above according to the second aspect of the present invention and at least one remote device hosting a blockchain, which at least one remote device being connected in communication to the device as described above according to the second aspect of the present invention.

[0019] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0020] Such a computer program may use any programming language, and be in the form of source code, object code, or an intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0021] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0022] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may include a storage means, such as a ROM, a CD-ROM, or a type ROM microelectronic circuit, or even a magnetic recording device or a hard drive.

[0023] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0024] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0025] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 3, in which:

[0026] [Fig.1] schematically illustrates a data communication environment relating to one or more vehicle collision tests, according to a particular and non-limiting embodiment of the present invention;

[0027] [Fig.2] schematically illustrates a device configured to generate content augmented reality graphic representing a vehicle collision test, according to a particular and non-limiting embodiment of the present invention;

[0028] [Fig.3] illustrates a flowchart of the different stages of a generation process of an augmented reality graphic content representative of a vehicle collision test, according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0029] A method and device for generating graphic content in augmented reality representative of a vehicle collision test will now be described in what follows with joint reference to Figures 1 to 3. The same elements are identified with the same reference signs throughout the description that follows.

[0030] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0031] According to a particular and non-limiting example of an embodiment of the present invention, the generation of graphic content in augmented reality, virtual reality or in Mixed reality, also known as AR graphic content (from the English "augmented reality" or in French "réalité augmentee"), for the simulation of a vehicle collision test is implemented for example by a computer or server type computing device, that is to say by one or more processors of the data processing device.

[0032] To this end, initial representative data from a three-dimensional video sequence of a vehicle crash test involving a crash dummy are received. This initial data may, for example, have been acquired by a set of cameras arranged outside and / or inside the vehicle to capture images of the dummy and the vehicle during the crash test. Advantageously, this initial data includes geometric information associated with the dummy and a set of elements forming the vehicle. This information is obtained, for example, by further processing of the initial data acquired by the cameras and / or sensors located, for example, on the dummy.

[0033] A digital twin, called a selected digital twin, is selected from a plurality of digital twins, each digital twin in the plurality of digital twins being representative of a three-dimensional model of a mannequin representing a person with a specific morphology. The digital twin is, for example, selected according to the desired morphology. Digital twins have, for example, been generated from statistical morphology data of a set of vehicle drivers to statistically represent a set of vehicle driver morphologies.

[0034] Second data representing a three-dimensional model of the mannequin associated with the selected digital twin are received, for example from a "cloud" server.

[0035] Finally, the AR graphic content is generated from the first and second data by replacing the mannequin in the three-dimensional video sequence with the three-dimensional model of the mannequin associated with the selected digital twin. Such AR graphic content corresponds, for example, to a set of images of the collision test from different viewpoints, this AR graphic content being intended to be rendered or played on an augmented reality graphic display device, for example a virtual reality headset or a CAVE.

[0036] Fig. 1 schematically illustrates a vehicle crash test data communication environment, according to a particular and non-limiting embodiment of the present invention.

[0037] The communication environment 1 includes, for example, a computing device 11 configured for the generation of AR graphic content, which corresponds, for example, to a computer or a server.

[0038] The computing device 11 is advantageously connected in communication with a system comprising, for example, one or more data processing devices 101, for example of the server type, and / or one or more remote devices 102 hosting one or more databases or ledger(s). This device or these devices 101, 102 belong, for example, to the "cloud" 100. This device or these devices 101, 102 constitute, for example, nodes of a network configured for hosting data or transactions of a blockchain formed of several blocks 110.

[0039] The data processing device(s) 101 and the remote device(s) 102, if applicable, are for example connected in communication with the "cloud" 100 via a wired link (for example according to Ethernet and / or via a fiber optic link) and / or via a wireless link of the Wifi® type (according to IEEE 802.11 or one of the variants of IEEE 802.11 or via a wireless link of the 4G or 5G cellular network type).

[0040] Each device 101, 102, 103 is thus configured to transmit data to the "cloud" 100 and to receive data from the "cloud" 100.

[0041] The communication infrastructure enabling wireless data communication between the computing device 11 and the "cloud" 100 and the device(s) 101, 102 hosted in the "cloud" 100 corresponds, for example, to a wired network infrastructure, a wireless cellular network infrastructure or a network infrastructure with wired and wireless components.

[0042] A wireless communication system enabling the exchange of data between the computing device 11 and the "cloud" 100 corresponds for example to a cellular network type communication system, for example an LTE (Long-Term Evolution) type network, LTE-Advanced (LTE-advanced) LTE 4G or 5G.

[0043] The communication environment 1 and the devices that compose it are configured for the implementation of a blockchain.

[0044] Blockchain, also called distributed ledger technology, is a digital tool for storing and communicating data without a central controlling authority. The blockchain is configured to allow blockchain users (in this case, the computing device 11 and the data processing device(s) 101 and / or the remote device(s) 102), connected via a network, to share data directly, without intermediaries. A blockchain is a ledger or large database that has the characteristic of being shared simultaneously by all its users, all of whom are also holders of this ledger and who all have the ability to record data on it, according to specific rules established by a computer protocol secured by cryptography.

[0045] In a blockchain, the identification of each part (for example the computing device 11 and / or the device(s) 101, 102) is carried out by a cryptographic process.

[0046] Each transaction is sent to a network (also called a storage “node”) of remote devices or computing units (e.g., computers or servers) located in different locations, for example in the “cloud” 100.

[0047] Each node hosts a copy of the database in which the history of transactions carried out is recorded. All stakeholders (i.e. for example the computing device 11 and / or the device(s) 101, 102) can access it simultaneously.

[0048] The security system relies on a consensus mechanism among all nodes for each addition of information or data. The data is decrypted and authenticated by data centers (also called "miners"). Each validated transaction is added to the database as a block (of the blockchain) of encrypted data, a block being able to contain several transactions.

[0049] Decentralizing security management prevents transaction tampering. Each new block added to the blockchain is linked to the previous one (except for the first block in the chain), and a copy of this new block is transmitted to all network nodes. Block integration is chronological, indelible, and tamper-proof.

[0050] Blockchain technology is known to those skilled in the art. For example, it is described in a summary report of the French National Assembly dated December 2018 and entitled "Report of the joint fact-finding mission on blockchain and its uses: a matter of sovereignty".

[0051] Terms relating to blockchain have also been standardized, in particular in the following standards: ISO 22739:2020, ISO / TR 23244:2020, ISO / TR 23455:2019 and ISO / TR 23576:2020.

[0052] A process for generating augmented reality graphic content, called AR graphic content, representing a vehicle collision test, for example based on blockchain technology 110, is implemented in the communication environment 1.

[0053] Some operations of the process are for example implemented by the computing device 11. Other operations of the process are for example implemented by one or more devices 101, 102 of the "cloud" 100.

[0054] In a first operation, first data representative of a three-dimensional (3D) video sequence of a collision test of the vehicle 10 carrying a mannequin (not shown in [Fig. 1]) are received by the computing device 11.

[0055] This first data is for example received from the "cloud" 100, this first data being hosted in a database 102. According to another example, this first data is received by one or more processors (or graphics processors, called GPU (from the English "Graphics Processing Unit" or in French "processeur graphique")) of a memory of the computing device 11, the first data being stored locally in the computing device 11.

[0056] The first data corresponds to a set of data acquired by a set of cameras, each camera in the set being configured to acquire images during a crash test of the vehicle 10 from a different viewpoint of the scene corresponding to the crash test. At least some of the cameras are configured to acquire images of the crash dummy during the crash test.

[0057] The cameras are thus arranged for example at different points in the passenger compartment of the vehicle 10. According to one variant, cameras are also arranged outside (on the vehicle 10 and / or along the path of the vehicle 10 associated with the crash test) to obtain images of the vehicle 10 and the dummy from other points of view (for example, points of view outside the vehicle 10).

[0058] According to one variant, several mannequins are carried in the vehicle 10. A mannequin represents a person in the vehicle 10 (or an occupant of the vehicle 10) corresponding for example to a driver of the vehicle 10 sitting in the driver's seat 10 (with the seat belt) and / or a passenger sitting in the front passenger seat of the vehicle 10 (with the seat belt).

[0059] Each crash test dummy is equipped with various sensors configured to measure the stresses or forces experienced by the dummy during the crash test. According to an optional variant, visual markers are placed at various locations on the dummy to allow the physical dummy to be replaced by a model, as described in the following steps.

[0060] A camera includes, for example, the following elements: - a photosensitive sensor corresponding for example to a matrix of photoreceptors associated for example with a Bayer filter; - an optical assembly arranged in front of the sensor with respect to the scene to be acquired by the sensor, the optical assembly comprising, for example, an arrangement of one or more lenses; and - optionally a depth sensor.

[0061] The first data advantageously includes geometric information associated with the mannequin and a set of elements forming the vehicle 10. This information is obtained for example by processing image data obtained from cameras, for example for object detection such as the mannequin. This information corresponds, for example, to finite element data, depth information, information relating to a mesh associated with the mannequin (or even to certain components of vehicle 10), information relating to control points of the mannequin obtained, for example, by the detection of markers positioned on the mannequin.

[0062] This information advantageously allows the mannequin in the 3D video content to be replaced by modeling data of another mannequin, as described below, according to all data processing methods known to a person skilled in the art.

[0063] Three-dimensional modeling of a mannequin corresponds, for example, to finite element modeling. According to other examples, the modeling corresponds to modeling by NURBS type curves (from the English "Non-Uniform Rational Basis Spline" or in French "B-spline rationnelle non uniforme"), parametric modeling, etc.

[0064] In a second operation, a digital twin is selected from a list comprising several digital twins, each digital twin being associated with a three-dimensional model of a mannequin representative of a person having a determined morphology.

[0065] The representative data of each three-dimensional model of a mannequin are, for example, hosted in the "cloud" and written into a block of the blockchain 110 in the form of a digital twin of the associated three-dimensional model of a particular mannequin.

[0066] Each digital twin thus represents a mannequin of particular morphology, for example a pregnant woman, a tall person, a person of average height, a short person, etc.

[0067] Digital twins are generated from statistical data relating to the drivers of a set of vehicles, for example, vehicles of the same brand. The data relating to the drivers is obtained, for example, during targeted surveys, with the permission of the drivers, making it possible to establish driver classes according to the morphological characteristics of the drivers, the drivers associated with the same class having the same morphology or the same type of morphology, a digital twin being generated for each class.

[0068] The selection of the digital twin is for example implemented via a graphical HMI (Human-Machine Interface).

[0069] In a third operation, second data representing the selected digital twin are transmitted by device 101 and received by the computing device 11.

[0070] This second data is transmitted, for example, via a wireless connection when the request was transmitted via a wireless connection. According to an alternative, this second data is transmitted via a wired connection, for example, when the request was transmitted via a wired connection.

[0071] In a fourth operation, the AR graphic content is generated from the first data and the second data by replacing the mannequin in the original three-dimensional video sequence with the three-dimensional model of the mannequin associated with the selected digital twin.

[0072] AR graphic content advantageously includes: - a set of images of the actual scene acquired during the crash test and representing different viewpoints of the scene, including the crash test dummy placed in vehicle 10 during the crash test; and - a virtual graphic object corresponding to the modeled mannequin of the selected digital twin displayed in superposition of the real images at the location of the physical mannequin placed in vehicle 10 during the collision test.

[0073] The embedding of the 3D model of the mannequin associated with the selected digital twin is implemented according to any data processing method known to the person skilled in the art, for example as described in the document entitled "Effective methods for embedding virtual objects in image sequences", by Gilles Simon and Marie-Odile Berger, published on October 19, 2006.

[0074] In a further operation, the generated AR graphic content is rendered or played on a display device, for example a holographic display device, a virtual reality headset, a monitor-type display screen, a display screen of a mobile communication device such as a tablet or a smartphone, from one or more viewpoints.

[0075] Such a process makes it possible to simulate or visualize in an immersive way, for example in 360°, the effects of the collision of the vehicle 10 on different mannequins of different types of morphology by simply selecting the associated digital twin.

[0076] This makes it possible to carry out safety tests of the crash test type on several scenarios with different types of mannequin in a fully digital manner to improve safety and comfort on board the vehicle 10 with a view of these digital simulations in immersion via the use of augmented reality while reducing the associated testing costs and time.

[0077] The data from digital twins is "shareable" via a blockchain-type platform, for example a private or consortium blockchain, in order to share this data transparently, securely and confidentiality in an immutable manner with other regulatory actors of EuroNCAP® for example.

[0078] Figure 2 schematically illustrates a device 2 configured for generating augmented reality graphic content representing a vehicle crash test, for example, vehicle 10, according to a particular and non-limiting embodiment of the present invention. Device 2 corresponds, for example, to the computing device 11. According to another embodiment, Device 2 corresponds to the processing device 101 or to a network node hosting the blockchain data, such as Device 102.

[0079] Device 2 is, for example, configured to carry out the operations described opposite [Fig. 1] and / or the steps of the process described opposite [Fig. 3]. Examples of such a device 2 include, but are not limited to, a smartphone, a tablet, a computer, a laptop, and a server. The elements of device 2, individually or in combination, can be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 2 can be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0080] The device 2 comprises one (or more) processor(s) 20 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 2. The processor 20 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 2 further comprises at least one memory 21, corresponding, for example, to volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0081] The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored on memory 21.

[0082] According to various particular and non-limiting embodiments, the device 2 is coupled in communication with other similar devices or systems, for example via a communication bus or through dedicated input / output ports.

[0083] According to a particular and non-limiting embodiment, the device 2 comprises a block 22 of interface elements for communicating with external devices. The interface elements of the block 22 comprise one or more of the following interfaces: - radio frequency RF interface, for example of the Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or of the Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or of the Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); - HDMI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French).

[0084] According to another particular and non-limiting embodiment, the device 2 includes a communication interface 23 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 230. The communication interface 23 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 230. The communication interface 23 corresponds, for example, to a wired Ethernet network (standardized by ISO / IEC 802-3).

[0085] According to a particular and non-limiting embodiment, the device 2 can provide output signals to one or more external devices, such as a display screen 240, touch or not, one or more speakers 250 and / or other peripherals 260 (projection system) via output interfaces 24, 25 and 26 respectively. According to a variant, one or more of the external devices is integrated into the device 2.

[0086] Figure 3 illustrates a flowchart of the different stages of a process of generation of augmented reality graphic content representative of a vehicle collision test, for example vehicle 10, according to a particular and non-limiting embodiment of the present invention. The method is implemented, for example, by one or more processors of a computing device such as device 11 or device 2 of [Fig. 2].

[0087] In a first step 31, first representative data of a three-dimensional video sequence of a vehicle crash test carrying a dummy are received, the first data including geometric information associated with the dummy and a set of elements forming the vehicle.

[0088] In a second step 32, a digital twin, called the selected digital twin, is selected from a plurality of digital twins, each digital twin in the plurality of digital twins being representative of a three-dimensional modeling of a mannequin representing a person with a specific morphology.

[0089] In a third step 33, second data representing a three-dimensional model of the mannequin associated with the selected digital twin are received.

[0090] In a fourth step 34, the augmented reality graphic content is generated from the first and second data by replacing the mannequin in the three-dimensional video sequence with the three-dimensional model of the mannequin associated with the selected digital twin

[0091] According to one variant, the variants and examples of the operations described in relation to [Fig.1] apply to the steps of the process in [Fig.3].

Claims

Demands

1. Method for generating augmented reality graphic content representative of a vehicle crash test (10), said method being implemented by at least one processor and comprising the following steps: - receiving (31) first data representative of a three-dimensional video sequence of a vehicle crash test (10) carrying a dummy, said first data comprising geometric information associated with said dummy and a set of elements forming said vehicle; - selecting (32) a digital twin, said selected digital twin, from a plurality of digital twins, each digital twin of the plurality of digital twins being representative of a three-dimensional model of a dummy representative of a person having a determined morphology;- reception (33) of second data representing a three-dimensional model of the mannequin associated with said selected digital twin; - generation (34) of said augmented reality graphic content from said first data and said second data by replacing said mannequin in said three-dimensional video sequence with the three-dimensional model of the mannequin associated with said selected digital twin, said plurality of digital twins being generated from statistical morphology data of a set of drivers of vehicles of the same brand of vehicles.

2. A method according to claim 1, further comprising a step of displaying said graphic content in augmented reality on a display device.

3. A method according to claim 1 or 2, wherein said first data are generated by composing image data received from a plurality of cameras configured for image acquisition of said collision test from a plurality of viewpoints.

4. A method according to any one of claims 1 to 3, wherein each digital twin of said plurality of digital twins is written in a blockchain (110).

5. A method according to any one of claims 1 to 4, wherein said geometric information and said second data comprise representative finite element data.

6. Computer program comprising instructions for carrying out the method according to any one of the preceding claims, when such instructions are executed by a processor.

7. A computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to any one of claims 1 to 5

8. 1 a J. Device (2) for generating augmented reality graphic content representative of a vehicle crash test, said device (2) comprising a memory (21) associated with at least one processor (20) configured for carrying out the steps of the method according to any one of claims 1 to 5.

9. Data communication system comprising the device (2) according to claim 8 and at least one remote device (101) hosting a blockchain (110), said at least one remote device (101) being connected in communication to said device (2), said system being configured for the implementation of the steps of the process according to any one of claims 1 to 5.

Images