A dynamic calibration model training method, dynamic calibration method, device and medium

Abstract

This application discloses a dynamic calibration model training method, dynamic calibration method, device, and medium, belonging to the field of data processing technology. The method mainly includes: annotating the vehicle's internal structure in images captured by an onboard camera to obtain the annotation location information of the areas where the vehicle's internal structure is located; obtaining an image stitching image based on the annotated areas; inputting the stitching image into a preset neural network model to obtain the location information of one or more points corresponding to the vehicle's internal structure; and using a loss function between the annotation location information and the point location information to obtain a dynamic calibration model between the vehicle and the onboard camera. By providing an integrated neural network model, the required relevant information can be obtained from a single input of the stitched image, improving system operating efficiency while ensuring data accuracy is not affected. This allows for fast and accurate acquisition of the extrinsic parameter matrix, providing accurate conditions for determining whether the driver is distracted.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a dynamic calibration model training method, dynamic calibration method, device and medium. Background Technology

[0002] In the field of autonomous driving, camera extrinsic calibration involves transforming the perception results in the camera coordinate system to the vehicle coordinate system. For example, by transforming the driver's line of sight to the vehicle coordinate system, we can know the driver's current line of sight and thus determine whether the driver is distracted while driving.

[0003] In existing technologies, methods for determining whether a driver is distracted while driving often involve placing a driver fatigue detection system (DMS) camera on the airbag cover or similar location. This static calibration method uses the DMS camera to collect real-time information about the driver's face to determine distraction. However, in this static calibration method, the location of the DMS camera differs from the driver's position. Different drivers have different heights and seat adjustments, leading to a discrepancy between the perceived direction of the driver's gaze and their actual gaze, potentially resulting in misjudgments. Summary of the Invention

[0004] To address the problems in existing technologies where dynamic calibration followed by judgment inevitably leads to reduced operating efficiency of the driver fatigue detection system, and the accuracy of camera extrinsic parameters also affects the accuracy of the judgment process of the driver fatigue detection system, this application mainly provides a dynamic calibration model training method, dynamic calibration method, device, and medium.

[0005] In a first aspect, embodiments of this application provide a dynamic calibration model training method, comprising: labeling the vehicle's internal structure in one or more images captured by an onboard camera, thereby obtaining the labeling location information corresponding to the labeled points in the region where the vehicle's internal structure is located; cropping regions in each image to obtain cropped regions corresponding to each vehicle's internal structure; stitching the cropped regions together to obtain a stitched image corresponding to the vehicle's internal structure; inputting the stitched image into a preset neural network model to obtain the location information of one or more points corresponding to each vehicle's internal structure; and adjusting the neural network model using a loss function between the labeled location information and the point location information to obtain a dynamic calibration model between the vehicle and the onboard camera.

[0006] Secondly, embodiments of this application provide a dynamic calibration method, comprising: selecting one or more regions enclosed by the vehicle's internal structure in the image based on real-time images of the vehicle's internal structure captured by onboard cameras mounted around the autonomous vehicle; inputting the image containing the region into a pre-established dynamic calibration model to obtain the position information of one or more points corresponding to the region; wherein the pre-establishment process of the dynamic calibration model includes: annotating the vehicle's internal structure in the image; obtaining the annotation position information corresponding to the annotation points in the region where the vehicle's internal structure is located; cropping the region in the image to obtain the cropped region corresponding to each vehicle's internal structure; stitching the cropped regions to obtain a stitched image corresponding to the vehicle's internal structure image; inputting the stitched image into a preset neural network model to obtain the position information of one or more points corresponding to each vehicle's internal structure; adjusting the neural network model using a loss function between the position information of the vehicle's internal structure and the position information of the points to obtain a dynamic calibration model; and obtaining the extrinsic parameter matrix between the vehicle and the onboard camera using the position information of the vehicle's internal structure and the position information of the points in a pre-established vehicle coordinate system.

[0007] Thirdly, embodiments of this application provide a dynamic calibration device, comprising: a region selection image module, which selects one or more regions enclosed by the vehicle's internal structure in real-time images captured by onboard cameras mounted around the autonomous vehicle; and a region location prediction module, which inputs the image containing the region into a preset dynamic calibration model to obtain the location information of one or more points corresponding to the region. The pre-establishment process of the dynamic calibration model includes: an image annotation module, which annotates the vehicle's internal structure in the image and obtains the annotation location information corresponding to the annotation points in the region where the vehicle's internal structure is located; and an image cropping module, which crops the regions in the image. The system performs cropping to obtain cropped regions corresponding to each vehicle's internal structure. An image stitching module stitches these cropped regions together to obtain a stitched image of the vehicle's internal structure. A position information acquisition module inputs the stitched image into a pre-defined neural network model to obtain position information for one or more points corresponding to each vehicle's internal structure. A model acquisition module uses a loss function between the position information of the vehicle's internal structure and the position information of the points to adjust the neural network model and obtain a dynamically calibrated model. Finally, an extrinsic parameter acquisition module uses the position information of the vehicle's internal structure and the position information of the points in a pre-established vehicle coordinate system to obtain the extrinsic parameter matrix between the vehicle and the onboard camera.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions. When the computer instructions are executed, they cause the computer to perform the dynamic calibration model training method or dynamic calibration method described above.

[0009] Fifthly, embodiments of this application provide a computer device including a processor and a memory, the memory storing computer instructions that are operated to execute the dynamic calibration model training method or dynamic calibration method in the above-described scheme.

[0010] The technical solution of this application provides an integrated neural network model that allows the required relevant information to be obtained by inputting a stitched image at once. This improves the system's operating efficiency while ensuring that the accuracy of the data is not affected, thereby quickly and accurately obtaining the extrinsic parameter matrix and providing accurate conditions for determining whether the driver is distracted. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an optional implementation of a dynamic calibration model training method according to this application;

[0013] Figure 2 This is a schematic diagram of an optional implementation of a dynamic calibration method according to this application;

[0014] Figure 3 This is a schematic diagram of an optional embodiment of a dynamic calibration device according to this application.

[0015] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0016] The preferred embodiments of this application will now be described in detail with reference to the accompanying drawings, so that the advantages and features of this application can be more easily understood by those skilled in the art, thereby providing a clearer and more definite definition of the scope of protection of this application.

[0017] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0018] In the field of autonomous driving, camera extrinsic calibration involves transforming the perception results in the camera coordinate system to the vehicle coordinate system. For example, by transforming the driver's line of sight to the vehicle coordinate system, we can know the driver's current line of sight and thus determine whether the driver is distracted while driving.

[0019] In existing technologies, methods for determining whether a driver is distracted while driving often involve placing a driver fatigue detection system (DMS) camera on the airbag cover or similar location. This static calibration method uses the DMS camera to collect real-time information about the driver's face to determine distraction. However, in this static calibration method, the position of the DMS camera differs from the driver's position. Therefore, the driver's gaze direction obtained using this method differs from the driver's actual gaze direction, leading to potential misjudgments.

[0020] To reduce the distance between the DMS camera's location and the driver's position, it was considered to place the DMS camera on the steering column of the vehicle. However, since the column can be adjusted by the user, the camera's extrinsic parameters need to be dynamically calibrated. Dynamic calibration before judgment will inevitably reduce the operating efficiency of the driver fatigue detection system, and the accuracy of the camera's extrinsic parameters will also affect the accuracy of the driver fatigue detection system's judgment process. To address the problems existing in the prior art, this application mainly provides a dynamic calibration model training method, dynamic calibration method, device, and medium, which includes: based on one or more images of the vehicle's internal structure captured by an onboard camera, annotating the images of the vehicle's internal structure to obtain the annotation position information corresponding to the annotation points in the region where the vehicle's internal structure is located; cropping the region in each image to obtain the cropped region corresponding to each vehicle's internal structure; stitching the cropped region images to obtain a stitched image corresponding to the vehicle's internal structure; inputting the stitched image into a preset neural network model to obtain information on one or more line points corresponding to each curve in the region; and adjusting the neural network model using a loss function between the vehicle's relevant data and the line point information to obtain a dynamic calibration model of the vehicle and the onboard camera.

[0021] By providing an integrated neural network model, the required relevant information can be obtained from a single input stitched image. This improves system efficiency while ensuring data accuracy, thus enabling the rapid and accurate acquisition of the extrinsic parameter matrix and providing accurate conditions for determining whether the driver is distracted.

[0022] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. The specific embodiments described below can be combined with each other to form new embodiments. The same or similar ideas or processes described in one embodiment may not be repeated in other embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0023] Figure 1 An alternative implementation of a dynamic calibration model training method according to this application is shown.

[0024] exist Figure 1 In the optional implementation shown, the dynamic calibration model training method mainly includes step S101, which involves annotating the vehicle's internal structure in one or more images taken by the vehicle-mounted camera to obtain the annotation position information corresponding to the annotation points in the area where the vehicle's internal structure is located.

[0025] In this optional embodiment, the vehicle-mounted camera includes a DMS camera with a driver fatigue detection system. Since the purpose of using this vehicle-mounted camera is to detect whether the driver is distracted while driving, the camera is oriented towards the inside of the vehicle, and the shooting range includes the interior of the vehicle, including the driver. In one optional embodiment, the camera can be mounted on the steering column. The driver's adjustment of the steering wheel can trigger the dynamic calibration process. Since the structure inside the vehicle is fixed and does not change due to steering wheel adjustments, the structure inside the vehicle can be used as a reference. During the dynamic calibration process, an image of the area where the vehicle's internal structure is located is first acquired. The area corresponding to the vehicle's internal structure in the image is manually labeled, and the position information corresponding to the labeled points in the area is recorded as the aforementioned labeling position information, ensuring the accuracy of the reference during the operation of this solution.

[0026] In an optional example of this application, since the dynamic calibration function is actually an action triggered by the driver adjusting the steering wheel, in order to ensure the diversity of data during data collection and to simulate the real state of the vehicle during driving, as many obvious curves as possible in the possible vehicle internal structure are selected. The aforementioned vehicle internal structure includes, but is not limited to, the area behind the sunroof, the left side of the sunroof, the window sealing strip, the roof, the intersection of the B-pillar, the intersection of the B-pillar and the roof, and the seat belt groove on the B-pillar. After data collection, the curves are manually annotated to ensure the accuracy of the curves and to provide conditions for subsequent calculation of accurate external parameter matrices using the curves.

[0027] In an optional embodiment of this application, after annotating the vehicle's internal structure in the image, the method further includes: performing data augmentation on the image and using the augmented image as part of the image.

[0028] In this optional embodiment, before manually annotating the image, data augmentation is performed on the image. For example, changes in noise, brightness, and contrast are used to simulate environmental changes, and augmented images under various environments are obtained. And / or, based on changes in angles and heights such as translation and rotation at the geometric level, augmented images under various states are obtained to simulate as many situations as possible that a vehicle may encounter during driving. The augmented images are then used as images for subsequent processing steps, so that when the stitched image is input into the neural network model, the robustness of the model is enhanced and the accuracy of the model is improved.

[0029] exist Figure 1 In the optional implementation shown, the dynamic calibration model training method further includes step S102, which involves cropping the regions in each image to obtain the cropping regions corresponding to the internal structures of each vehicle.

[0030] In this optional embodiment, since this solution utilizes the area where the vehicle's internal structure is located to perform various operations in order to obtain a dynamic calibration model, the areas in the image that are unrelated to the vehicle's internal structure are cropped out to obtain the cropped area corresponding to the vehicle's internal structure. This allows subsequent operations to be performed only on the cropped area, improving the computation speed and reducing the impact of other factors on the accuracy of the dynamic calibration model in this solution.

[0031] exist Figure 1 In the optional implementation shown, the dynamic calibration model training method further includes step S103, which involves stitching the cropped area to obtain a stitched image corresponding to the vehicle interior structure image.

[0032] In this optional implementation, after cropping the image, the cropped areas are stitched together to obtain a stitched image, saving the time of dynamic calibration. This allows the stitched image to be input into the neural network model at once to obtain the results, eliminating the need to input the cropped areas or images one by one to obtain the results, thus speeding up the system operation.

[0033] In one optional embodiment of this application, stitching the cropped area to obtain a stitched image corresponding to the vehicle's internal structure further includes: scaling the cropped area according to a preset width so that the width of the cropped area is consistent with the preset width; and stitching the scaled cropped area to obtain a stitched image.

[0034] In this optional embodiment, the image width is preset, i.e., a pre-defined width. Each cropping region is scaled according to the preset width so that the width of all cropping regions is consistent with the preset width. The cropping regions are then stitched together according to the dimension where the width is located to obtain a stitched image. The width of the stitched image is the preset width, which can be set according to the size of the image transmission channel, provided that the preset width is not greater than the width of the image transmission channel. Obtaining the stitched image in this solution allows for rapid acquisition of the corresponding relationship when subsequently converting the labeled reference objects to the vehicle coordinate system or camera coordinate system, thus accelerating the system's operating speed.

[0035] In an optional embodiment of this application, for cropping region 1 (50 pixels * 60 pixels), cropping region 2 (70 pixels * 120 pixels), and cropping region 3 (20 pixels * 30 pixels), if the preset width is 60 pixels, then cropping region 1 does not need to be scaled. The width of cropping region 2 is reduced to half of its original size, making the size of cropping region 2 70 pixels * 60 pixels. The width of cropping region 3 is increased to twice its original size, making the size of cropping region 3 20 pixels * 60 pixels. This ensures that the widths of the three cropping regions are consistent with the preset width, so that the three cropping regions can be stitched together according to the dimension where the width is located, resulting in a stitched image with a width of 60 pixels and a height of 140 pixels.

[0036] In one optional embodiment of this application, stitching the cropped area to obtain a stitched image corresponding to the vehicle's internal structure further includes: scaling the cropped area according to a preset height so that the height of the cropped area is consistent with the preset height; and stitching the scaled cropped area to obtain a stitched image.

[0037] In this optional embodiment, the width and height of the image are preset. Each cropping region is scaled according to the preset width so that the height of all cropping regions is consistent with the preset height. The cropping regions are then stitched together according to the dimension where the height is located to obtain a stitched image. The height of the stitched image is the preset height, which is set according to the size of the image transmission channel and should not exceed the height of the image transmission channel. Obtaining the stitched image in this solution allows for rapid acquisition of the corresponding relationship when subsequently converting the labeled reference objects to the vehicle coordinate system or camera coordinate system, thus accelerating the system's operating speed.

[0038] In an optional embodiment of this application, for cropping region 1 (60 pixels * 50 pixels), cropping region 2 (120 pixels * 70 pixels), and cropping region 3 (30 pixels * 20 pixels), if the preset height is 60 pixels, then cropping region 1 does not need to be scaled. The height of cropping region 2 is reduced to half of its original size, making the size of cropping region 2 60 pixels * 70 pixels. The height of cropping region 3 is increased to twice its original size, making the size of cropping region 3 60 pixels * 20 pixels. This ensures that the height of the three cropping regions is consistent with the preset height, so that the three cropping regions can be stitched together according to the dimension where the height is located, resulting in a stitched image with a height of 60 pixels and a width of 140 pixels.

[0039] In one optional embodiment of this application, stitching the cut areas to obtain a stitched image corresponding to the vehicle's internal structure further includes: scaling the cut areas according to a preset channel size so that the size of the cut areas is consistent with the preset channel size; and stitching the scaled cut areas to obtain a stitched image.

[0040] In this optional embodiment, since the vehicle-mounted cameras used for calibration are mostly infrared cameras, the number of image transmission channels is 1. The width and height of the image transmission channel can be used as the preset channel size, or the preset channel size can be set to be less than or equal to the width and height of the image transmission channel. This ensures that the scaled cropped area can be completely transmitted through the image transmission channel after stitching, guaranteeing the accuracy of the cropped area information and providing a foundation for obtaining an accurate dynamic calibration model in the future.

[0041] In an optional embodiment of this application, for cropping region 1 (60 pixels * 50 pixels), cropping region 2 (120 pixels * 100 pixels), and cropping region 3 (30 pixels * 25 pixels), if the height of the preset channel size is 60 pixels and the width is 50 pixels, then cropping region 1 does not need to be scaled. The height and width of cropping region 2 are both reduced to half of their original size, making the size of cropping region 2 60 pixels * 100 pixels. The height and width of cropping region 3 are both enlarged to twice their original size, making the size of cropping region 3 60 pixels * 100 pixels. This ensures that the height and width of the three cropping regions are consistent with the height and width in the preset channel size, so that the three cropping regions can be superimposed, resulting in a stitched image with a height of 60 pixels and a width of 50 pixels.

[0042] exist Figure 1 In the specific implementation shown, the dynamic calibration model training method further includes step S104, which involves inputting the stitched image into a preset neural network model to obtain the position information of one or more points corresponding to the internal structure of the vehicle.

[0043] In this optional implementation, since the curves corresponding to the vehicle's internal structure in the image have already been manually labeled, the stitched image includes the curves corresponding to the vehicle's internal structure. The stitched image is then input into the neural network model to obtain the position information of the points on the curves corresponding to the vehicle's internal structure, providing a foundation for obtaining an accurate dynamic calibration model in the future.

[0044] In an optional embodiment of this application, the stitched image is input into a preset neural network model to obtain the position information of one or more points corresponding to the internal structure of the vehicle, which further includes: obtaining the contour curve of the internal structure of the vehicle according to the neural network model; and obtaining the position information of one or more points on the contour curve according to the contour curve.

[0045] In this optional embodiment, after the stitched image is input into the neural network model, the neural network model extracts the curves in the stitched image, namely the contour curves; then, the position information of one or more points on the contour curves is obtained based on the contour curves.

[0046] In an optional embodiment of this application, obtaining the position information of one or more points on the contour curve based on the contour curve further includes: selecting N equally divided points of the contour curve as the position information of one or more points on the contour curve.

[0047] In an optional embodiment of this application, obtaining the position information of one or more points on the contour curve based on the contour curve further includes: selecting N equally divided points of the contour curve as the position information of one or more points on the contour curve, where N is a natural number that is not 0.

[0048] In this optional embodiment, the stitched image corresponding to the region is input into a neural network model to obtain N equal division points of the curve. These N equal division points are used as the position information of the points corresponding to the curve. The curve can be a curve corresponding to the internal structure of the vehicle, such as the curve on the left side of the antenna or the curve of the B-pillar. Using the division points as points on the curve facilitates obtaining the 3D information corresponding to the position information of the points on the curve in the vehicle coordinate system.

[0049] In one optional embodiment of this application, the vehicle interior structure corresponding to the contour curve includes one or more of the following: the area behind the sunroof, the left side of the sunroof, the window sealing strip, the roof, the intersection of the B-pillar, the intersection of the B-pillar and the roof, and the seat belt groove on the B-pillar.

[0050] In this optional embodiment, since the internal structure of the vehicle includes, but is not limited to, the intersection of the sunroof rear, the sunroof left side, the window sealing strip, the roof, the B-pillar intersection, the B-pillar and roof intersection, and the seat belt groove on the B-pillar, the contour curves are the curves corresponding to the sunroof rear, the sunroof left side, the window sealing strip, the roof, the B-pillar intersection, the B-pillar and roof intersection, and the seat belt groove on the B-pillar, respectively. Since the vehicle internal structure in the stitched image may not be able to completely capture all the internal structures of the vehicle, the contour curves in this solution include one or more curves corresponding to the vehicle internal structure.

[0051] In one optional embodiment of this application, the stitched image is input into a preset neural network model to obtain the position information of one or more points corresponding to the internal structure of the vehicle, and the method further includes: obtaining the confidence level of the points in the neural network model based on the one or more points corresponding to the internal structure of the vehicle.

[0052] In this optional embodiment, the stitched image is input into the neural network model to obtain the N equal division points of the curve and the confidence scores corresponding to the N equal division points. The N equal division points are taken as the points corresponding to the curve. The aforementioned confidence scores are the confidence scores corresponding to the points on the curve. The curve can be the curve corresponding to the internal structure of the vehicle, such as the curve on the left side of the antenna or the curve of the B-pillar. The obtained confidence scores provide a basis for the accuracy of the subsequently obtained dynamic calibration model.

[0053] exist Figure 1 In the specific implementation shown, the dynamic calibration model training method further includes step S105, which uses the loss function between the calibration location information and the point location information to adjust the neural network model and obtain the dynamic calibration model between the vehicle and the vehicle-mounted camera.

[0054] In this optional implementation, a loss function is calculated between the calibration position information and the position information of its corresponding point. Based on the loss function, the relevant parameters in the neural network model are adjusted to make the loss function smaller and smaller. When the loss function is less than or equal to a preset loss threshold, the neural network model at this time is determined to be a dynamic calibration model. In this solution, since fixed objects such as vehicle structural components that do not change due to the adjustment of the steering column are used as reference objects, the obtained dynamic calibration model has high accuracy.

[0055] In one optional embodiment of this application, the neural network model is adjusted using a loss function between the calibration location information and the point location information to obtain a dynamic calibration model between the vehicle and the vehicle-mounted camera. The method further includes: filtering out the point location information based on confidence level; and adjusting the neural network model using a loss function between the calibration location information and the filtered point location information to obtain a dynamic calibration model.

[0056] In this optional embodiment, before converting the position information of the points on the curve to the camera coordinate system, the points on the curve are filtered according to the confidence scores obtained above. Then, the filtered points on the curve are distorted and converted to the camera coordinate system to improve the accuracy of the dynamic calibration model.

[0057] Figure 2 An optional implementation of a dynamic calibration method according to this application is shown.

[0058] exist Figure 2In the optional implementation shown, the dynamic calibration method mainly includes: step S201, selecting one or more regions enclosed by the vehicle's internal structure in the image based on real-time images of the vehicle's internal structure captured by onboard cameras mounted around the autonomous vehicle; step S202, inputting the image containing the region into a pre-established dynamic calibration model to obtain the position information of one or more points corresponding to the region, wherein the pre-establishment process of the dynamic calibration model includes: step S2021, annotating the vehicle's internal structure in the image and obtaining the annotation position information corresponding to the annotation points in the region where the vehicle's internal structure is located; step S2022, processing the region in the image... In step S2023, the cropped areas are obtained for each internal structure of the vehicle. In step S2024, the cropped areas are stitched together to obtain a stitched image corresponding to the internal structure of the vehicle. In step S2025, the stitched image is input into a preset neural network model to obtain the position information of one or more points corresponding to each internal structure of the vehicle. In step S2026, the neural network model is adjusted using the loss function between the position information of the internal structure of the vehicle and the position information of the points to obtain a dynamic calibration model. In step S207, the external parameter matrix between the vehicle and the on-board camera is obtained using the position information of the internal structure of the vehicle and the position information of the points in the pre-established vehicle coordinate system.

[0059] In this optional implementation, during the actual driving of the autonomous vehicle, the onboard camera mounted on the vehicle captures images of the vehicle's internal structure in real time. The current image is input into a pre-established dynamic calibration model, which performs operations such as image annotation, stitching, and information extraction. Finally, the model outputs accurate 2D point position information corresponding to the vehicle's internal structure. Based on the correspondence between the 3D point position information of the vehicle's internal structure in the pre-established vehicle coordinate system and the aforementioned 2D information, the extrinsic parameter matrix between the vehicle and the onboard camera is calculated. The model's image annotation, stitching, and information extraction operations are similar in principle and technical effect to the dynamic calibration model training method described in any of the preceding embodiments, and will not be repeated here.

[0060] The dynamic calibration method provided in this application can be used to perform the dynamic calibration model training method described in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0061] Figure 3 An optional embodiment of a dynamic calibration device according to this application is shown.

[0062] exist Figure 3In the optional embodiment shown, the dynamic calibration device mainly includes: a region selection module 301, which selects one or more regions enclosed by the vehicle's internal structure in the image based on real-time images of the vehicle's internal structure captured by onboard cameras mounted around the autonomous vehicle; and a region location prediction module 302, which inputs the image containing the region into a preset dynamic calibration model to obtain the location information of one or more points corresponding to the region. The pre-establishment process of the dynamic calibration model includes: an image annotation module 3021, which annotates the vehicle's internal structure in the image and obtains the annotation location information corresponding to the annotation points in the region where the vehicle's internal structure is located; and an image cropping module 3022, which crops the region in the image... The system performs cropping on the internal structure of the vehicle to obtain cropped regions corresponding to each internal structure. Image stitching module 3023 stitches the cropped regions to obtain a stitched image corresponding to the internal structure of the vehicle. Position information acquisition module 3024 inputs the stitched image into a preset neural network model to obtain position information of one or more points corresponding to the internal structure of the vehicle. Model acquisition module 3025 adjusts the neural network model using a loss function between the labeled position information and the position information to obtain a dynamic calibration model. Extrinsic parameter acquisition module 303 uses the position information of the internal structure of the vehicle and the position information of the points in a pre-established vehicle coordinate system to obtain the extrinsic parameter matrix between the vehicle and the on-board camera.

[0063] In an optional embodiment of this application, the functional modules of the dynamic calibration device of this application may be directly in hardware, in software modules executed by a processor, or in a combination of both.

[0064] Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in this art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium.

[0065] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, but alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. Alternatively, the storage medium can be integrated with the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in the user terminal. Alternatively, the processor and storage medium can reside as discrete components in the user terminal.

[0066] The dynamic calibration device provided in this application can be used to execute the dynamic calibration model training method described in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0067] In another optional embodiment of this application, a computer-readable storage medium is provided, which stores computer instructions that are operated to perform the dynamic calibration model training method or dynamic calibration method described in the above embodiments.

[0068] In one optional embodiment of this application, a computer device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer instructions executable by the at least one processor, and the at least one processor operates the computer instructions to execute the dynamic calibration model training method or dynamic calibration method described in the above embodiments.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A dynamic model calibration training method, characterized in that, The method comprises the following steps: According to one or more images of the vehicle interior structure taken by the vehicle-mounted camera, manually label the vehicle interior structure in the image, and obtain the label position information corresponding to the label points of the region where the vehicle interior structure is located; Crop each region in the image respectively to obtain the corresponding cropped region of each vehicle interior structure; and splice the cropped regions to obtain a spliced image corresponding to the vehicle interior structure in the image; Input the spliced image into a preset neural network model to obtain the position information of one or more points corresponding to the vehicle interior structure; The method further comprises the following steps: According to the neural network model, obtain the contour curve of the vehicle interior structure, and obtain the position information of the one or more points on the contour curve; Adjust the neural network model using the loss function between the label position information and the position information of the points to obtain a dynamic calibration model of the vehicle-mounted camera, and output accurate position information of the points corresponding to the vehicle interior structure from the dynamic calibration model; And use the position information of the vehicle interior structure and the position information of the points in the pre-established vehicle body coordinate system to obtain the extrinsic matrix between the vehicle and the vehicle-mounted camera.

2. The dynamic calibration model training method of claim 1, wherein, The method further comprises the following steps: Scale the cropped regions according to a preset width so that the width of the cropped regions is consistent with the preset width; Splice the scaled cropped regions to obtain the spliced image.

3. The dynamic calibration model training method of claim 1, wherein, The method further comprises the following steps: Scale the cropped regions according to a preset height so that the height of the cropped regions is consistent with the preset height; Splice the scaled cropped regions to obtain the spliced image.

4. The dynamic calibration model training method of claim 1, wherein, The method further comprises the following steps: Scale the cropped regions according to a preset channel size so that the size of the cropped regions is consistent with the preset channel size; Splice the scaled cropped regions to obtain the spliced image.

5. The dynamic calibration model training method of claim 1, wherein, The method further comprises the following steps: After labeling the vehicle interior structure in the image, perform data augmentation on the image, and use the augmented image as the image.

6. The dynamic calibration model training method of claim 1, wherein, The method further comprises the following steps: Select N equal points of the contour curve as the position information of the one or more points on the contour curve, wherein N is a natural number not equal to 0.

7. The dynamic calibration model training method of claim 6, wherein The contour curve corresponding to the vehicle interior curve includes one or more of a rear sunroof, a left sunroof, a window seal, a roof, an intersection of a B column and the roof, an intersection of the B column and the roof, and a safety belt slot on the B column.

8. The dynamic calibration model training method according to any one of claims 1-7, characterized in that, The inputting the spliced image into the preset neural network model to obtain the position information of one or more points corresponding to the vehicle interior structure respectively further includes: In the neural network model, the confidence of the one or more points corresponding to the vehicle interior structure respectively is obtained.

9. The dynamic calibration model training method of claim 8, wherein, The adjusting the neural network model by using a loss function between the labeled position information and the position information of the points to obtain a dynamic calibration model between the vehicle and the vehicle-mounted camera further includes: According to the confidence, the position information of the points is filtered out; The adjusting the neural network model by using a loss function between the labeled position information and the position information of the filtered points to obtain the dynamic calibration model.

10. A dynamic calibration method, characterized by, It includes: According to the image with the vehicle interior structure captured by the vehicle-mounted camera loaded around the autonomous vehicle in real time, one or more regions enclosed by the vehicle interior structure are selected in the image; the image containing the regions is input into a pre-established dynamic calibration model to obtain the position information of one or more points corresponding to the regions, wherein the pre-establishment process of the dynamic calibration model includes manually labeling the vehicle interior structure in the image to obtain the labeled position information of the labeled points in the region where the vehicle interior structure is located, cropping the regions in the image to obtain the cropped regions corresponding to the vehicle interior structure respectively, splicing the cropped regions to obtain a spliced image corresponding to the vehicle interior structure image, inputting the spliced image into a preset neural network model to obtain the position information of one or more points corresponding to the vehicle interior structure respectively, wherein inputting the spliced image into the preset neural network model to obtain the position information of one or more points corresponding to the vehicle interior structure respectively specifically includes, According to the neural network model, the contour curve of the vehicle interior structure is obtained, and the position information of the one or more points on the contour curve is obtained according to the contour curve; and adjusting the neural network model by using a loss function between the labeled position information and the position information of the points to obtain the dynamic calibration model; and obtaining an external parameter matrix between the vehicle and the vehicle-mounted camera by using the position information of the vehicle interior structure in the pre-established vehicle body coordinate system and the position information of the points.

11. A dynamic calibration device, characterized by It includes: The region selection image module selects one or more regions enclosed by the vehicle interior structure in the image according to the image with the vehicle interior structure captured by the vehicle-mounted camera loaded around the autonomous vehicle in real time; The region position prediction module inputs the image containing the regions into a preset dynamic calibration model to obtain the position information of one or more points corresponding to the regions, wherein the pre-establishment process of the dynamic calibration model includes, An image labeling module, manually labeling the vehicle interior structure in the image, and obtaining labeled position information corresponding to a labeled point of a region where the vehicle interior structure is located, An image cropping module, cropping the region in the image, and obtaining a respective cropped region corresponding to each vehicle interior structure, An image splicing module, splicing the cropped regions, and obtaining a spliced image corresponding to the vehicle interior structure image, a position information obtaining module, inputting the spliced image into a preset neural network model, and obtaining position information of one or more points respectively corresponding to the vehicle interior structure, wherein the inputting the spliced image into the preset neural network model and obtaining the position information of the one or more points respectively corresponding to the vehicle interior structure specifically include, obtaining a contour curve of the vehicle interior structure according to the neural network model, and obtaining position information of the one or more points on the contour curve according to the contour curve; and A model obtaining module, adjusting the neural network model by using a loss function between the labeled position information and the position information of the points, and obtaining the dynamic calibration model; and An extrinsic parameter obtaining module, obtaining an extrinsic matrix between the vehicle and the vehicle-mounted camera by using position information of the vehicle interior structure and the position information of the points in a pre-established vehicle body coordinate system.

12. A computer-readable storage medium storing computer instructions, wherein, The computer instructions are operated to perform the dynamic calibration model training method of any one of claims 1-9 or the dynamic calibration method of claim 10.

Images