Parking environment analysis method and system, computer equipment and storage medium
By combining multiple small models and multimodal large models in parking environment analysis, the problem of difficulty in identifying dangerous situations in complex parking environments in a timely manner is solved in the existing technology, and the rapid and accurate parking environment information processing is achieved, and the system is improved robustness and recognition accuracy.
Patent Information
- Application Number
- CN202411024053.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
Smart Images

Figure CN118953366A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of vehicle technology, and in particular, relates to a parking environment analysis method, system, computer device and storage medium. Background Art
[0002] With the rapid development of intelligent driving technology, the analysis and perception of the vehicle's surrounding environment during parking has become an important factor in ensuring vehicle safety and preventing accidents. Existing parking environment analysis methods are mainly based on traditional image processing algorithms or deep learning-based methods. However, the environment around the vehicle during parking is complex and changeable, and objects such as people, cars, and pets may pose a threat to the safety of the vehicle or passengers. Existing methods have limitations in dealing with complex and changeable parking environments, and it is difficult to identify potential dangerous situations in a timely manner. Summary of the invention
[0003] The embodiments of the present application provide a parking environment analysis method, system, computer device and storage medium, which can solve the technical problems that existing methods have limitations when dealing with complex and changeable parking environments and are difficult to identify potential dangerous situations in a timely manner.
[0004] In a first aspect, an embodiment of the present application provides a parking environment analysis method, comprising:
[0005] Input the acquired vehicle camera data into multiple small models to obtain multiple small model output results;
[0006] The output results of the plurality of small models and the vehicle-mounted camera data are input into a multimodal large model to generate parking environment information; wherein the model scale of the small model is smaller than that of the multimodal large model.
[0007] In a possible implementation manner of the first aspect, the step of inputting the acquired vehicle-mounted camera data into a plurality of small models to obtain a plurality of small model output results further includes:
[0008] Inputting the vehicle-mounted camera data into the multimodal large model to obtain an initial recognition result;
[0009] When the initial recognition result meets the preset abnormal condition, the step of inputting the acquired vehicle-mounted camera data into a plurality of small models to obtain output results of a plurality of small models is executed.
[0010] In a possible implementation of the first aspect, inputting the vehicle-mounted camera data into the multimodal large model to obtain an initial recognition result includes:
[0011] The vehicle-mounted camera data and preset questions are input into the multimodal large model to obtain an initial recognition result.
[0012] In a possible implementation of the first aspect, the parking environment information includes: at least two of object category, object position, object distance, and object dangerous action;
[0013] and / or,
[0014] The multiple small models include at least two of a segmentation model, a detection model, and a dangerous action recognition model, and the corresponding multiple small model output results include at least two of an object category, an object position and distance, and an object dangerous action recognition result.
[0015] In a possible implementation of the first aspect, inputting the output results of the plurality of small models and the vehicle-mounted camera data into a multimodal large model to generate parking environment information includes:
[0016] Based on the multimodal large model, text processing is performed on the output results of the multiple small models to obtain word vectors after position encoding;
[0017] Extracting features from the vehicle-mounted camera data to obtain visual features;
[0018] The word vector and the visual feature are concatenated, and the concatenated combined vector is input into a decoder in the multimodal large model to generate the parking environment information.
[0019] In a possible implementation manner of the first aspect, before concatenating the word vector and the visual feature, the method further includes:
[0020] Perform feature alignment on the visual features to obtain visual features aligned with the word vectors.
[0021] In a possible implementation manner of the first aspect, the method further includes:
[0022] The parking environment information and the vehicle-mounted camera data are sent.
[0023] In a second aspect, an embodiment of the present application provides a parking environment analysis system, including:
[0024] An output module is used to input the acquired vehicle-mounted camera data into multiple small models to obtain multiple small model output results;
[0025] An information generation module is used to input the output results of the multiple small models and the vehicle-mounted camera data into the multimodal large model to generate parking environment information; wherein the model scale of the small model is smaller than that of the multimodal large model.
[0026] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the parking environment analysis method described in any one of the first aspects above is implemented.
[0027] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the parking environment analysis method described in any one of the first aspects above is implemented.
[0028] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the parking environment analysis method described in any one of the above-mentioned first aspects.
[0029] Compared with the prior art, the embodiment of the present application has the following beneficial effects: first, the acquired vehicle camera data is input into multiple small models. Due to the use of multiple small models, multiple small model output results can be obtained quickly, in real time, and flexibly. Afterwards, the output results of multiple small models and the vehicle camera data are input into a multi-modal large model together. By processing the multi-modal data, various complex environments can be processed to generate accurate and reliable parking environment information.
[0030] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0032] Figure 1 This is a schematic diagram of an application scenario of a parking environment analysis method provided by an embodiment of the present application;
[0033] Figure 2 It is a flowchart of a parking environment analysis method provided by an embodiment of the present application;
[0034] Figure 3 is a flow chart of a parking environment analysis method provided by another embodiment of the present application;
[0035] Figure 4 is a flow chart of a parking environment analysis method provided in yet another embodiment of the present application;
[0036] Figure 5 is a flow chart of a parking environment analysis method provided by another embodiment of the present application;
[0037] Figure 6 This is a schematic diagram of an application scenario of a parking environment analysis method provided by another embodiment of the present application;
[0038] Figure 7 is a structural diagram of a parking environment analysis system provided in an embodiment of the present application;
[0039] Figure 8 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0040] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0041] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0042] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0043] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0044] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0045] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0046] The parking environment analysis method provided in this application can be applied to Figure 1 In the application environment shown, the vehicle-mounted camera data acquisition device 101 is used to obtain vehicle-mounted camera data. The parking environment analysis system 102 can run a preset computer program to analyze and process the obtained vehicle-mounted camera data to obtain parking environment information.
[0047] Optionally, the vehicle-mounted video data acquisition device 101 may be a camera installed on the vehicle. The camera may include a fisheye lens, a panoramic camera, a night vision camera, a thermal imaging camera, a high dynamic range camera, a depth camera, etc., and is not limited to the above cameras. Optionally, the camera may be installed around the vehicle so that videos and images can be collected from all directions.
[0048] Optionally, the vehicle-mounted camera data acquisition device 101 may also be a roadside device, which acquires the vehicle-mounted camera data and sends the acquired vehicle-mounted camera data to the parking environment analysis system 102 for analysis and processing.
[0049] Optionally, the parking environment analysis system 102 can be a local system, which is deployed locally to achieve rapid response and meet application scenarios with high real-time requirements. Optionally, the parking environment analysis system 102 can also be partially deployed locally and partially deployed on a server, which can be implemented as an independent server or a server cluster composed of multiple servers, so as to meet application scenarios such as complex analysis and large data volumes.
[0050] Existing parking environment analysis methods are mainly based on traditional image processing algorithms or deep learning-based methods. Traditional image processing algorithms usually rely on pixel values for calculations, and changes in illumination will directly affect pixel values, resulting in deviations in algorithm results. For example, under strong lighting conditions, the shadow of an object will become darker, causing the algorithm to be unable to accurately identify the boundaries of the object. In addition, traditional image processing algorithms are usually calculated based on local features. When an object is obscured, local feature information will be lost, causing the algorithm to be unable to accurately identify the object. For objects that move quickly or drastically, there may be image blur, making feature extraction difficult, resulting in reduced algorithm recognition accuracy. For deep learning-based methods, in an open world, the environment is complex and changeable, and scenes and situations that have never appeared in the training data may appear, causing the model to perform poorly in these scenes, resulting in bad cases, that is, special situations that the model cannot handle correctly.
[0051] To solve the above problems, this application adopts a solution combining multiple small models and a multi-modal large model to analyze the parking environment.
[0052] Figure 2 A schematic flow chart of a parking environment analysis method provided in the present application is shown. As an example but not a limitation, the method can be applied to the above-mentioned parking environment analysis system 102.
[0053] S201, input the acquired vehicle-mounted camera data into multiple small models to obtain multiple small model output results.
[0054] The vehicle-mounted camera data refers to the video and image information captured by the camera installed on the vehicle. For the relevant description of the camera, please refer to Figure 1 The illustrated embodiments will not be described in detail here.
[0055] Among them, the small model can be a lightweight model, a specific task model, or a simplified model. A lightweight model refers to a deep learning model that is relatively small in terms of model size, computational complexity, and memory usage. A specific task model is obtained by training for a specific task, such as a segmentation model, a detection model, a dangerous action recognition model, etc. A simplified model is a model generated by simplifying a large model or a common model, such as reducing the number of model layers, reducing the number of model parameters, etc., in order to reduce the computational complexity and memory usage of the model.
[0056] Among them, each small model corresponds to a small model output result.
[0057] In the embodiment of the present application, due to the collaborative processing of multiple small models, different features can be learned, the generalization ability of the model can be improved, and the occurrence of bad cases can be reduced. Due to the use of small models, the amount of calculation is smaller, the reasoning speed is faster, and predictions can be made faster to meet the needs of real-time applications, such as autonomous driving, real-time monitoring, etc. The combination of multiple small models can reduce the risk of failure of a single model and improve the robustness of the system.
[0058] S202, inputting the output results of the plurality of small models and the vehicle-mounted camera data into a multimodal large model to generate parking environment information; wherein the model scale of the small model is smaller than that of the multimodal large model.
[0059] Among them, a multimodal big model can refer to a big model that can process and understand multiple types of data (i.e., multimodal data). It is no longer limited to a specific data set and can process multiple types of data such as text, images, audio, and video at the same time.
[0060] Specifically, the multimodal large model receives a large amount of multi-type data from different fields during training. Through cross-modal integration and learning of these large amounts of multi-type data, it learns rich semantic information from them, and exhibits new features and new functions that are beyond what a single modal combination can achieve. It has strong emergence and generalization capabilities, provides more accurate and reliable judgments, and overcomes the limitations of traditional deep learning methods in complex environments.
[0061] The parking environment information may be a summary generated based on a multimodal large model.
[0062] The output results of multiple small models and the vehicle-mounted camera data are input into the multimodal large model together, and accurate and reliable parking environment information is generated by processing the multimodal data.
[0063] Among them, the above-mentioned small model and multimodal large model can be deployed locally, belonging to local models. Alternatively, the small model and the multimodal large model can be deployed on the server, belonging to an online model. Alternatively, the small model is deployed locally, belonging to a local model, and the multimodal large model is deployed on the server, belonging to an online model. Optionally, multiple small models can also be partially deployed locally and partially deployed on the server. The deployment method can be set according to actual needs.
[0064] In the embodiment of the present application, the acquired vehicle camera data is first input into multiple small models. Due to the use of multiple small models, multiple small model output results can be obtained quickly, in real time, and flexibly. Afterwards, the output results of multiple small models and the vehicle camera data are input into the multi-modal large model together. By processing the multi-modal data, various complex environments can be processed to generate accurate and reliable parking environment information.
[0065] In the prior art, a small model is usually used for the initial analysis, and when the initial analysis results determine that there is an abnormality, a secondary analysis is performed. For example, when a small model is used to analyze that there is a person near the vehicle, in this case, a detailed analysis is performed again to obtain a more accurate abnormality. However, the use of a small model for the initial analysis may result in information loss, inaccurate analysis results, and failure to timely and accurately identify potential dangers. In one possible implementation, in order to improve accuracy and avoid information loss, and to more accurately identify potential dangers, before the acquired vehicle-mounted camera data is input into multiple small models, it can be analyzed using a large multimodal model. When the analysis results are abnormal, the steps of the scheme in the above embodiment are executed. The specific implementation method is as follows: Figure 3 , Figure 3 A schematic flow chart of another parking environment analysis method provided by the present application is shown.
[0066] S301, input the vehicle-mounted camera data into the multimodal large model to obtain an initial recognition result.
[0067] The model architecture of the multimodal large model is the same as that of the multimodal large model in the above embodiment, but the input data of the two models are different. In the embodiment of the present application, the vehicle-mounted camera data and the preset questions are input into the multimodal large model to obtain the initial recognition result.
[0068] Among them, the initial recognition result is the result obtained by using a multimodal large model to perform preliminary analysis and recognition of the vehicle-mounted camera data. If the initial recognition result meets the preset abnormal conditions, it means that there is a high probability of an abnormal situation, and it is necessary to further analyze the parking environment to output an accurate analysis result. If the initial recognition result does not meet the preset abnormal conditions, it means that there is no abnormal situation, and there is no need to further analyze the parking environment. Among them, because the vehicle-mounted camera data is preliminarily analyzed and recognized using a multimodal large model, potential abnormal or dangerous situations can be identified more timely and accurately, greatly improving the scene coverage of abnormal recognition. And when there is no abnormal or dangerous situation, avoid frequent calls to the multimodal large model to generate parking environment information, thereby reducing the waste of resources.
[0069] The initial recognition result may be a binary result. Optionally, the binary result includes but is not limited to any one of the following expressions: the presence of an abnormality or the absence of an abnormality, 1 or 0, yes or no, etc.
[0070] For ease of understanding, the initial recognition result is taken as yes or no as an example. In an embodiment of the present application, several questions can be pre-set. For example, is it possible for objects around the current vehicle to touch the vehicle? Are there any dangerous actions around the current vehicle that may threaten the safety of the vehicle? Does the current parking environment of the vehicle have an impact on the safety of the vehicle? The multimodal large model analyzes the on-board camera data, and judges according to the questions, and outputs a "yes" or "no" field. Among them, the initial recognition result can exist in two forms. The first is to output a "yes" or "no" field for each question. If there are 3 questions, the model outputs 3 fields, and the 3 fields can be "yes" or "no". The second is to judge according to the question and obtain a "yes" or "no" field, and then analyze the obtained fields. If there is at least one field that is "yes", the model outputs a result of "yes", and if all fields are "no", the model outputs a result of "no".
[0071] It should be noted that the above-mentioned preset questions are only some illustrative examples. Related questions not recorded in the embodiments of this application are also within the scope of this application. In specific implementation, several questions will be set according to actual needs, and several questions can be combined, changed, etc. according to the actual parking environment.
[0072] In one possible implementation, the vehicle camera data and the preset questions are input into the multimodal large model to obtain the initial recognition results, including:
[0073] Based on the multimodal large model, text processing is performed on preset questions to obtain word vectors after position encoding; feature extraction is performed on the vehicle-mounted camera data to obtain visual features; the word vectors and visual features are spliced, and the combined vector obtained after splicing is input into the decoder in the multimodal large model to generate an initial recognition result.
[0074] Optionally, before concatenating the word vectors and the visual features, the visual features need to be aligned to obtain visual features aligned with the word vectors. Specifically, the visual features and the text features (ie, the word vectors) are aligned.
[0075] Optionally, multiple questions can be combined into a single text, which is used as input to a large multimodal model.
[0076] Among them, text processing of text can also be called encoding processing, which specifically includes: first converting the integrated text into a vector representation, each vector representation corresponds to a token. Then the token can be converted into a word embedding vector using word embedding and other methods. At the same time, the token is positionally encoded to generate a position encoding vector to represent the position information of each token. The position encoding vector has the same dimension as the word embedding vector. Afterwards, the position encoding vector and the word embedding vector are combined (e.g., added or concatenated) to obtain the above-mentioned position-encoded word vector.
[0077] In addition to processing the text, the vehicle-mounted camera data also needs to be processed and analyzed. Specifically, a visual feature extractor can be used to extract features. The visual feature extractor here can be a Vision Transformer (ViT), a convolutional neural network CNN, and the like.
[0078] In an embodiment of the present application, word vectors and visual features are concatenated, and the combined vector obtained after concatenation is input into a decoder in a multimodal large model, so that accurate initial recognition results can be output.
[0079] Optionally, after acquiring the vehicle-mounted camera data, the vehicle-mounted camera data can be preprocessed, including but not limited to: size adjustment, dark light enhancement, image denoising, image sharpening, data enhancement (rotating, flipping, scaling, etc. of the image to increase data diversity), etc.
[0080] S302, when the initial recognition result meets the preset abnormal condition, the acquired vehicle-mounted camera data is input into multiple small models to obtain multiple small model output results.
[0081] Among them, an abnormal condition is preset, and the abnormal condition is a trigger condition for executing the small model analysis. For example, in the first initial recognition result of S301, the preset abnormal condition is: judging whether there is at least one field in the initial recognition result containing multiple fields that is "yes", if there is at least one field that is "yes", then the initial recognition result meets the preset abnormal condition. For example, in the second initial recognition result of S301, the preset abnormal condition is: judging whether the field of the initial recognition result is "yes", if it is "yes", then the initial recognition result meets the preset abnormal condition.
[0082] The acquired vehicle-mounted camera data is input into a plurality of small models to obtain a description of the output results of the plurality of small models, which is consistent with the description in S201 and will not be described in detail here.
[0083] S303, inputting the output results of the plurality of small models and the vehicle-mounted camera data into a multimodal large model to generate parking environment information.
[0084] The relevant description of S303 is consistent with the relevant description of S202 and will not be repeated here.
[0085] In an embodiment of the present application, when performing parking environment analysis, the vehicle-mounted camera data is first analyzed for the first time through a multimodal large model to obtain an initial recognition result. The subsequent steps are performed only when the initial recognition result meets the preset abnormal conditions. The data that needs to be subsequently analyzed is screened through the multimodal large model and the preset abnormal conditions, thereby reducing unnecessary calculations and improving efficiency. In addition, when the initial recognition result meets the preset abnormal conditions, a plurality of small model output results are first obtained based on the analysis of the small model. Since a plurality of small models are used, the output results of the plurality of small models can be obtained quickly, in real time, and flexibly. Afterwards, the output results of the plurality of small models and the vehicle-mounted camera data are input into the multimodal large model together. By processing the multimodal data, various complex environments can be processed to generate accurate and reliable parking environment information.
[0086] Among them, as mentioned above, the model architecture of the multimodal large model in S301 and the multimodal large model in S303 is the same, and the processing flow of the model input is the same. However, the input data and output results of the two models are different. And the functions or capabilities of the two models are different. The multimodal large model in S301 is used for initial judgment and anomaly detection, and the multimodal large model in S303 is used for information generation. The multimodal large model in S301 determines whether there is an abnormal situation based on the on-board camera data. When there is an abnormal situation, the multimodal large model in S303 is used to generate complete, rich and accurate information; when there is no abnormal situation, no subsequent analysis will be performed.
[0087] Among them, in the first embodiment of the present application (i.e., the embodiment composed of S201 and S202), the multimodal large model is not used for initial judgment first. In this case, regardless of whether there is an abnormal situation, whether the abnormal situation is serious, etc., the multimodal large model will be directly used to generate complete, rich and accurate information.
[0088] In a possible implementation, the multiple small models include at least two of a segmentation model, a detection model, and a dangerous action recognition model, and the corresponding output results of the multiple small models include at least two of an object category, an object position and distance, and an object dangerous action recognition result. Optionally, the output result corresponding to the detection model may also include an object color.
[0089] Among them, the segmentation model divides the video or image into different regions, marks the category of each region, identifies objects around the vehicle, such as roads, pedestrians, animals, etc., and marks their categories.
[0090] The detection model identifies objects around the vehicle in the video or image and determines the bounding box of the object. Each bounding box contains the object's category, location, distance, color, confidence, etc.
[0091] The dangerous action recognition model is used to identify and predict possible dangerous actions in images or videos, such as fast-moving vehicles and pedestrians crossing the road.
[0092] It should be noted that only the small model described above is described in the embodiments of the present application. However, other small models may also be involved in the parking information analysis, such as trajectory prediction model, speech recognition model, etc. Related small models not described in the embodiments of the present application are also within the scope of the present application.
[0093] In the embodiment of the present application, due to the collaborative processing of multiple small models, different features can be learned, the generalization ability of the model can be improved, and the occurrence of bad cases can be reduced. Due to the use of small models, the amount of calculation is smaller, the reasoning speed is faster, and predictions can be made faster to meet the needs of real-time applications, such as autonomous driving, real-time monitoring, etc. The combination of multiple small models can reduce the risk of failure of a single model and improve the robustness of the system.
[0094] In a possible implementation, the parking environment information includes: at least two of object category, object position, object distance, and object dangerous action. Optionally, in addition to the above information, the parking environment information may also include object color, object age, and other information, and is not limited to the above information.
[0095] In this embodiment, due to the use of multiple small models, multiple small model output results are obtained based on multi-dimensional or multi-attention levels. The multiple small model output results are combined with the on-board camera data again to obtain rich, comprehensive and accurate parking environment information, allowing users to accurately understand potential risks.
[0096] In one possible implementation, Figure 4 A schematic flow chart of another parking environment analysis method provided by the present application is shown.
[0097] S401, input the vehicle-mounted camera data into the multimodal large model to obtain an initial recognition result.
[0098] Among them, the relevant description of S401 in the embodiment of the present application is consistent with the relevant description of S301 in the above embodiment, and will not be repeated here.
[0099] S402, when the initial recognition result meets the preset abnormal condition, the acquired vehicle-mounted camera data is input into multiple small models to obtain multiple small model output results.
[0100] Among them, the relevant description of S402 in the embodiment of the present application is consistent with the relevant description of S302 in the above embodiment, and will not be repeated here.
[0101] S403: Based on the multimodal large model, text processing is performed on the output results of the multiple small models to obtain word vectors after position encoding.
[0102] Before the text processing of the output of the small model based on the multimodal large model, the output of the small model needs to be integrated into a text, which is used as the input of the multimodal large model.
[0103] Among them, the text processing of the output results of the integrated small model can also be called encoding processing, which specifically includes: first converting the output results of the integrated small model into a vector representation, each vector representation corresponds to a token. Then, the token can be converted into a word embedding vector using methods such as word embedding. At the same time, the token is positionally encoded to generate a position encoding vector to represent the position information of each token. The position encoding vector has the same dimension as the word embedding vector. Afterwards, the position encoding vector and the word embedding vector are combined (such as addition or concatenation) to obtain the above-mentioned position-encoded word vector.
[0104] S404: extract features from the vehicle-mounted camera data to obtain visual features.
[0105] In an embodiment of the present application, the integrated small model output results and the vehicle-mounted camera data are input into the multimodal large model together. While the small model output results are being text-processed, the vehicle-mounted camera data also needs to be processed and analyzed.
[0106] Specifically, features may be extracted using a visual feature extractor, which may be a Vision Transformer (ViT), a convolutional neural network (CNN), or the like.
[0107] In a possible implementation, S405 is executed to perform feature alignment on the visual features to obtain visual features aligned with the word vectors.
[0108] Specifically, after extracting features using a visual feature extractor and obtaining visual features, the visual features and text features (i.e., word vectors) are aligned. The feature alignment may be implemented by attention-based alignment or projection-based alignment.
[0109] Among them, the alignment based on the attention mechanism can use the cross attention mechanism to pay attention to the visual features and text features, and learn the corresponding relationship between them. For example, using the self-attention mechanism of ViT, the visual features and text features are taken as input, and the attention weights between them are calculated to achieve alignment. Alternatively, using multimodal attention, the visual features and text features are simultaneously input into the multimodal attention mechanism, the relationship between them is learned, and the aligned features are generated.
[0110] Alignment based on the projection matrix can be achieved by projecting the visual features and text features into the same space using a linear projection matrix, such as using a shared linear layer to project the visual features and text features into the same dimension. Alternatively, a nonlinear projection function can be used to project the visual features and text features into the same space to achieve alignment, such as using a neural network to map the visual features and text features into the same dimension.
[0111] It should be noted that this application only describes alignment based on the attention mechanism and alignment based on the projection matrix, which are only used as illustrative examples. This application may also adopt other existing feature alignment methods, and related feature alignment methods not described in the embodiments of this application are also within the scope of this application.
[0112] S406, concatenating the word vector and the visual feature, and inputting the concatenated combined vector into a decoder in the multimodal large model to generate the parking environment information.
[0113] Among them, the splicing methods may include: simple splicing, weighted splicing, attention splicing or gated splicing, etc.
[0114] Simple concatenation directly connects the features of different modalities to form a longer vector. In the embodiment of the present application, the above word vectors and visual features are directly connected in series.
[0115] Weighted concatenation assigns different weights to features of different modalities, and then performs weighted summation to form a new vector. The weights can be obtained through training and learning, or can be set based on experience. In the embodiment of the present application, different weights are assigned to the above-mentioned word vectors and visual features, and then weighted summation is performed to form a new vector.
[0116] Attention mechanism, use the attention mechanism to learn the relationship between different modal features, assign different weights according to the relationship, and then perform weighted summation. For example, the cross attention mechanism can be used to learn the correspondence between word vectors and visual features, and assign different weights according to the correspondence.
[0117] Gating mechanism: Use gating mechanism to select important information from different modal features and integrate them together. For example, you can use Gated Recurrent Unit (GRU) to learn the relationship between word vectors and visual features, and select important information for splicing based on the relationship.
[0118] It should be noted that this application only describes simple splicing, weighted splicing, attention splicing or gated splicing, which are only illustrative examples. This application may also adopt other splicing methods, and related splicing methods not described in the embodiments of this application are also within the scope of this application.
[0119] In the prior art, there are limitations to using a single modal model to perform parking environment analysis. Using only a text model to generate parking environment information will be limited by the text description. Using only a visual model to generate parking environment information will be limited by the image content and will not provide sufficient semantic information. In the embodiments of the present application, parking environment analysis is performed based on text features and visual features. Text features and visual features are complementary. Text features provide semantic information, and visual features provide visual information. Fusion of the two can provide more comprehensive information. A large multimodal model can better understand the data, improve the model's predictive power and accuracy, and generate more accurate environmental information.
[0120] In a possible implementation, S407 is executed to send the parking environment information and the vehicle-mounted camera data.
[0121] The parking environment information and the vehicle-mounted camera data may be sent to at least one of the following: a mobile terminal, a vehicle-mounted display screen, a navigation system, and a cloud platform.
[0122] In an embodiment of the present application, after obtaining the parking environment information, the parking environment information and the vehicle-mounted camera data can be sent or displayed. For example, the parking environment information and the vehicle-mounted camera data can be sent to the owner's mobile phone APP, or the parking environment information and the vehicle-mounted camera data can be displayed on the vehicle display screen. The parking environment information can also be converted into a voice signal, and the parking environment information is broadcast by voice. The owner can understand the parking environment in advance, choose a suitable parking location, and view the surrounding environment through the vehicle-mounted camera data to avoid accidents such as collisions. Alternatively, the parking environment information and the vehicle-mounted camera data are sent to the navigation system to help the navigation system provide more accurate route planning and parking guidance services. Alternatively, the parking environment information and the vehicle-mounted camera data are uploaded to the cloud platform, and data analysis and mining can be performed, such as analyzing the usage of the parking lot, identifying potential traffic safety hazards, etc.
[0123] In a possible implementation, after the parking environment information is generated, since the parking environment information is a summary, keywords may be extracted from the parking environment information, and the parking risk level may be determined based on at least one extracted keyword.
[0124] Among them, natural language processing technology can be used to extract keywords.
[0125] In a possible implementation, a first mapping relationship between keywords and risk levels is pre-established, and then the risk level is determined based on at least one extracted keyword, including: determining the risk level corresponding to each keyword based on the first mapping relationship; and determining the final risk level based on at least one risk level. If there is only one keyword and only one risk level is determined, the risk level is determined as the final risk level; if there are multiple keywords corresponding to multiple risk levels, the highest risk level among the multiple risk levels can be determined as the final risk level.
[0126] In one possible implementation, a second mapping relationship from keywords to risk scores is pre-established, and a third mapping relationship between the overall risk score and the risk level is established; the risk level is determined based on at least one extracted keyword, including: determining the risk score corresponding to each keyword based on the second mapping relationship; summing up the risk scores to obtain an overall risk score; and determining the risk level corresponding to the overall risk score based on the third mapping relationship.
[0127] In one possible implementation, a weighted sum is performed based on the risk score to obtain an overall risk score, including: obtaining the frequency of keyword occurrence and determining a weight factor based on the frequency; and performing a weighted sum based on the weight factor of each keyword and the risk score of the keyword to obtain an overall risk score.
[0128] In a possible implementation, determining the risk level based on the at least one extracted keyword may also include: inputting the extracted keyword into a preset risk assessment model to output the risk level.
[0129] In a possible implementation, when the initial recognition result meets the preset abnormal condition, a first risk score can be determined based on the preset question; a second risk score can be determined based on the parking environment information; an overall risk score can be determined based on the first risk score and the second risk score; and a risk level corresponding to the overall risk score can be determined. For the description of the risk score, see above.
[0130] Among them, determining the first risk score based on preset questions may include: each question corresponds to a binary result, and the first risk score is determined based on multiple binary results corresponding to multiple preset questions. For ease of understanding, the binary result of yes or no is used as an example. If there are 3 questions, the corresponding binary results are "yes", "yes", and "no". Then the number of "yes" is divided by the total number of questions to determine the first risk score, that is, the first risk score is 2 / 3. Among them, the above examples are only illustrative examples, and other means can also be used to obtain risk scores. Similar means are all within the scope of protection of this application.
[0131] In order to facilitate the understanding of the process of combining multiple small models and a multi-modal large model to process vehicle-mounted camera data, the attached Figure 5 The schematic flow chart of the parking environment analysis method shown in FIG. 1 is used to describe in detail. Among them, the vehicle-mounted camera data data1 of the time period t1 is first obtained, the vehicle-mounted camera data data1 is preprocessed, and then the preprocessed vehicle-mounted camera data data1 is input into the multimodal large model to obtain the initial recognition result. If the initial recognition result does not meet the preset abnormal condition, the vehicle-mounted camera data is re-acquired. If the initial recognition result meets the preset abnormal condition, the vehicle-mounted camera data of the current time period is processed, and the vehicle-mounted camera data of the current time period can be the vehicle-mounted camera data data1 previously acquired, or the vehicle-mounted camera data data1 of the time period t1+the vehicle-mounted camera data data2 of the time period t2, wherein t2 is the preset time period after t1, and the current time period is t1+t2. Both t1 and t2 can be set according to actual needs. Optionally, the vehicle-mounted camera data of the current time period can be preprocessed, such as denoising, to obtain clearer vehicle-mounted camera data. Then, the vehicle-mounted camera data is analyzed using multiple small models to obtain multiple small model output results. The output results of the small model are integrated into a text, which is used as the input of the multimodal large model. After the above processing, the integrated text and the processed vehicle camera data of the current time period are input into the multimodal large model together, and the text and vehicle camera data are processed at the same time, wherein the text is processed to obtain the position-encoded word vector. Feature extraction is performed on the vehicle camera data, and the extracted features are aligned to obtain visual features aligned with the word vector, which are then spliced with the word vector and the spliced vector is input into the decoder to generate parking environment information, that is, generate a summary.
[0132] In order to facilitate the understanding of the scheme, Figure 6 The application scenario diagram of the parking environment analysis method shown in the figure is used as an example. This example is only for illustration and not for limitation. Figure 6 As shown, the vehicle-mounted camera data are 5 images.
[0133] For image a, after being processed by multiple small models and the results integrated, the resulting text is "In the currently known picture: there are vehicles near your car; there are people near your car; the vehicles near your car have large door movements; the picture is on the left side of your car. Please generate a sentence description of this picture." The parking environment information output by the multimodal large model is "When a person in white clothes opens the passenger door of a white car, he may hit your car."
[0134] For image b, after being processed by multiple small models and the results integrated, the resulting text is "In the currently known image: there are vehicles near your car; there are people near your car; the picture is behind your car. Please generate a sentence description of this image." The parking environment information output by the multimodal large model is "a person in black pants is standing behind your car, and a white car with an open trunk is parked behind your car."
[0135] For image c, after being processed by multiple small models and the results integrated, the resulting text is "In the currently known picture: there are vehicles near your car; there are people near your car; the picture is to the left of your car. Please generate a sentence description of this picture." The parking environment information output by the multimodal large model is "A lady in a blue top rides a two-wheeled vehicle passing by the left side of your car."
[0136] For image d, after being processed by multiple small models and the results integrated, the resulting text is "In the currently known image: there are people near the car; there are pets near the car; the picture is on the right side of the car. Please generate a sentence description of this image." The parking environment information output by the multimodal large model is "A lady in a white top is walking a black dog and a white dog past the right front of the car, and there is a pile of garbage on the right side of the car."
[0137] For image e, after being processed by multiple small models and the results integrated, the resulting text is "It is currently known that in the picture: there is someone near your car; someone may take dangerous actions against your car; the picture is on the right side of your car. Please generate a sentence description of this picture." The parking environment information output by the multimodal large model is "a person in black clothes is standing on the right side of your car and may hit your car."
[0138] It can be seen from the output information of this application scenario that based on the above-mentioned solution of this application, various complex environments can be handled to generate accurate and reliable parking environment information.
[0139] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0140] Corresponding to the parking environment analysis method described in the above embodiment, Figure 7 A structural block diagram of a parking environment analysis system provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0141] Reference Figure 7 , the parking environment analysis system includes:
[0142] The output module 701 is used to input the acquired vehicle-mounted camera data into multiple small models to obtain multiple small model output results;
[0143] The information generation module 702 is used to input the output results of the multiple small models and the vehicle-mounted camera data into the multimodal large model to generate parking environment information; wherein the model scale of the small model is smaller than that of the multimodal large model.
[0144] In a possible implementation, the parking environment analysis system further includes:
[0145] The initial recognition module is used to input the vehicle-mounted camera data into the multimodal large model to obtain an initial recognition result; when the initial recognition result meets a preset abnormal condition, the output module 701 is executed.
[0146] In a possible implementation, the initial identification module is used to:
[0147] The vehicle-mounted camera data and preset questions are input into the multimodal large model to obtain an initial recognition result.
[0148] In a possible implementation manner, the parking environment information includes: at least two of: object category, object position, object distance, and object dangerous action.
[0149] In one possible implementation, the multiple small models include at least two of a segmentation model, a detection model, and a dangerous action recognition model, and the corresponding multiple small model output results include at least two of an object category, an object position and distance, and an object dangerous action recognition result.
[0150] In a possible implementation, the information generating module 702 is used to:
[0151] Based on the multimodal large model, text processing is performed on the output results of the multiple small models to obtain word vectors after position encoding;
[0152] Extracting features from the vehicle-mounted camera data to obtain visual features;
[0153] The word vector and the visual feature are concatenated, and the concatenated combined vector is input into a decoder in the multimodal large model to generate the parking environment information.
[0154] In a possible implementation, the information generating module 702 is used to:
[0155] Perform feature alignment on the visual features to obtain visual features aligned with the word vectors.
[0156] In a possible implementation, the parking environment analysis system further includes:
[0157] A sending module is used to send the parking environment information and the vehicle-mounted camera data.
[0158] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0159] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0160] An embodiment of the present application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.
[0161] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0162] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0164] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0165] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0166] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0168] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
[0169] Figure 8 This is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. Figure 8 As shown, the computer device of this embodiment includes: at least one processor 20 ( Figure 8 Only one is shown in the figure), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein when the processor 20 executes the computer program 22, the steps in any of the above-mentioned parking environment analysis method embodiments are implemented.
[0170] The computer device may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 8 It is only an example of a computer device and does not constitute a limitation of the computer device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.
[0171] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0172] In some embodiments, the memory 21 may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 21 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory 21 may also include both an internal storage unit of the computer device and an external storage device. The memory 21 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is to be output.
[0173] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.
[0174] The user personal information processed by the applicant will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, which may involve the user's account information, device information, driving information, vehicle information or other related information. The applicant will treat the user's personal information and its processing with a high degree of diligence.
[0175] The Applicant attaches great importance to the security of user personal information and has adopted reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.
Claims
1. A parking environment analysis method, characterized in that: include: Input the acquired vehicle camera data into multiple small models to obtain multiple small model output results; The output results of the plurality of small models and the vehicle-mounted camera data are input into a multimodal large model to generate parking environment information; wherein the model scale of the small model is smaller than that of the multimodal large model.
2. The parking environment analysis method according to claim 1, characterized in that: The method of inputting the acquired vehicle-mounted camera data into a plurality of small models to obtain a plurality of small model output results also includes: Inputting the vehicle-mounted camera data into the multimodal large model to obtain an initial recognition result; When the initial recognition result meets the preset abnormal condition, the step of inputting the acquired vehicle-mounted camera data into a plurality of small models to obtain output results of a plurality of small models is executed.
3. The parking environment analysis method according to claim 2, characterized in that: The step of inputting the vehicle-mounted camera data into the multimodal large model to obtain an initial recognition result includes: The vehicle-mounted camera data and preset questions are input into the multimodal large model to obtain an initial recognition result.
4. The parking environment analysis method according to claim 1, characterized in that: The parking environment information includes: at least two of object category, object location, object distance, and object dangerous action; and / or, The multiple small models include at least two of a segmentation model, a detection model, and a dangerous action recognition model, and the corresponding multiple small model output results include at least two of an object category, an object position and distance, and an object dangerous action recognition result.
5. The parking environment analysis method according to claim 1, characterized in that: The step of inputting the output results of the plurality of small models and the vehicle-mounted camera data into the multi-modal large model to generate parking environment information includes: Based on the multimodal large model, text processing is performed on the output results of the multiple small models to obtain word vectors after position encoding; Extracting features from the vehicle-mounted camera data to obtain visual features; The word vector and the visual feature are concatenated, and the concatenated combined vector is input into a decoder in the multimodal large model to generate the parking environment information.
6. The parking environment analysis method according to claim 5, characterized in that: The concatenating the word vector and the visual feature further includes: Perform feature alignment on the visual features to obtain visual features aligned with the word vectors.
7. The parking environment analysis method according to claim 1, characterized in that: The method further comprises: The parking environment information and the vehicle-mounted camera data are sent.
8. A parking environment analysis system, characterized in that: include: An output module is used to input the acquired vehicle-mounted camera data into multiple small models to obtain multiple small model output results; An information generation module is used to input the output results of the multiple small models and the vehicle-mounted camera data into the multimodal large model to generate parking environment information; wherein the model scale of the small model is smaller than that of the multimodal large model.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
