Visual positioning method and device, equipment and storage medium
The multimodal pre-training model automatically marks the damaged area of the vehicle, which solves the problem of cumbersome vehicle image marking in visual positioning tasks, improves efficiency and stability, and reduces manual intervention.
Patent Information
- Application Number
- CN202411240395.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-03
AI Technical Summary
The marking process of vehicle images in the current visual positioning task is cumbersome, relies on manual marking, is time-consuming and labor-intensive, and is easily affected by subjective factors, resulting in low marking efficiency and stability.
Using multimodal pre-training model, by obtaining the features and text features of preset vehicle images, combining color sets, generating predicted colors, and using cross entropy or mean square error loss functions to train the model, automatically marking the damaged area of the vehicle, reducing manual labor Intervention.
Improve the efficiency and stability of vehicle image marking, reduce marking time, avoid the influence of subjective factors, and realize automated vehicle damaged area marking.
Smart Images

Figure CN119068439A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of artificial intelligence and financial technology, and in particular to visual positioning methods, devices, equipment and storage media. Background Art
[0002] Visual Grounding is a cross-cutting task in the fields of computer vision and natural language processing. With the development of science and technology, visual grounding has brought more convenience to people's lives and work.
[0003] However, the labeling process of the current vehicle image in the current visual positioning task is cumbersome, which is not conducive to improving the labeling efficiency of the current vehicle image. The reason is that the existing technology mainly uses manual labeling to label the current vehicle image, and the manual labeling method requires a large amount of data comparison, which is not only time-consuming and labor-intensive, but also easily affected by subjective factors, and is therefore not conducive to improving the labeling efficiency of the current vehicle image. Summary of the invention
[0004] The present invention provides a visual positioning method, device, computer equipment and storage medium to solve the technical problem that the marking process of the current vehicle image in the current visual positioning task is cumbersome and is not conducive to improving the marking efficiency of the current vehicle image.
[0005] In a first aspect, a visual positioning method is provided, comprising:
[0006] Obtain a preset vehicle image in a preset visual positioning task, obtain a preset bounding box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and obtain a preset color set, wherein the color set includes a plurality of preset colors;
[0007] Acquire preset image features and preset text features corresponding to the preset damaged area of the vehicle;
[0008] According to the preset image feature, the preset text feature, the color set and a predefined acquisition method, obtaining a probability value of each of the preset colors output by the multimodal pre-training model;
[0009] Generate scores for the preset colors according to the probability values of the preset colors and a predefined score model, and select the preset color with the highest score as the predicted color;
[0010] Obtaining a loss value between the predicted color and an actual color marking the preset damaged area of the vehicle, training the multimodal pre-training model with the goal of reducing the loss value;
[0011] A current vehicle image in a current visual positioning task is obtained, a current color output by the trained multimodal pre-trained model is obtained, and on the current vehicle image, a damaged area of the current vehicle is marked with the current color to generate a marked current vehicle image.
[0012] Furthermore, the obtaining of preset image features and preset text features corresponding to the preset damaged area of the vehicle includes:
[0013] Performing feature extraction on the preset bounding box to obtain preset image features corresponding to the preset damaged area of the vehicle;
[0014] Feature extraction is performed on the preset text prompt information to obtain preset text features corresponding to the preset vehicle damaged area.
[0015] Further, the obtaining of the probability value of each of the preset colors output by the multimodal pre-trained model according to the preset image feature, the preset text feature, the color set and the predefined acquisition method includes:
[0016] Based on a cross-modal attention mechanism, the preset image features, the preset text features, and the color set are input into a multimodal pre-training model;
[0017] Obtain probability values of each of the preset colors output by the multimodal pre-trained model based on the preset image features, the preset text features, and the color set.
[0018] Furthermore, generating scores for each preset color according to the probability values of each preset color and a predefined score model, and selecting the preset color with the highest score as the predicted color, includes:
[0019] Obtaining a preset storage path, and importing the probability value of each preset color through the storage path;
[0020] According to the probability values of the respective preset colors and a predefined score model, scores of the respective preset colors are generated, and the preset color with the highest score is selected as the predicted color.
[0021] Further, the obtaining of the loss value between the predicted color and the actual color of the preset vehicle damaged area marked, with the goal of reducing the loss value, training the multimodal pre-training model, includes:
[0022] Obtaining a loss function, wherein the loss function includes one of a cross entropy loss function and a mean square error loss function;
[0023] The loss value between the predicted color and the actual color of the preset vehicle damaged area marked is obtained through the cross entropy loss function or the mean square error loss function, and the multimodal pre-training model is trained with the goal of reducing the loss value.
[0024] Furthermore, the step of obtaining a current vehicle image in a current visual positioning task, obtaining a current color output by the trained multimodal pre-trained model, marking a damaged area of the current vehicle on the current vehicle image using the current color, and generating a marked current vehicle image includes:
[0025] During the training process of the multimodal pre-training model, when the loss value is less than a preset value, the training of the multimodal pre-training model is stopped, and the trained multimodal pre-training model is saved;
[0026] A current vehicle image in a current visual positioning task is obtained, a current vehicle damaged region of the current vehicle image is obtained, the current vehicle damaged region is input into the trained multimodal pre-trained model, a current color output by the trained multimodal pre-trained model based on the current vehicle damaged region is obtained, and on the current vehicle image, the current vehicle damaged region is marked with the current color to generate the marked current vehicle image.
[0027] Furthermore, in the step of obtaining the current vehicle image in the current visual positioning task, obtaining the current color output by the trained multimodal pre-trained model, marking the damaged area of the current vehicle on the current vehicle image with the current color, and generating the marked current vehicle image, the visual positioning method includes:
[0028] A preset insurance system is connected, and the marked current vehicle image is sent to the insurance system.
[0029] In a second aspect, a visual positioning device is provided, comprising:
[0030] A first acquisition module is used to acquire a preset vehicle image in a preset visual positioning task, acquire a preset boundary box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and acquire a preset color set, wherein the color set includes a plurality of preset colors;
[0031] A second acquisition module is used to acquire preset image features and preset text features corresponding to the preset damaged area of the vehicle;
[0032] A determination module, configured to obtain a probability value of each of the preset colors output by the multimodal pre-trained model according to the preset image feature, the preset text feature, the color set, and a predefined acquisition method;
[0033] A selection module, used to generate scores for each preset color according to the probability values of each preset color and a predefined score model, and select the preset color with the highest score as the predicted color;
[0034] A training module, used for obtaining a loss value between the predicted color and an actual color marking the preset damaged area of the vehicle, and training the multimodal pre-training model with the goal of reducing the loss value;
[0035] A generation module is used to obtain a current vehicle image in a current visual positioning task, obtain a current color output by the multimodal pre-trained model that has been trained, mark a damaged area of the current vehicle on the current vehicle image with the current color, and generate a marked current vehicle image.
[0036] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned visual positioning method when executing the computer program.
[0037] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned visual positioning method are implemented.
[0038] The present application provides a visual positioning method, apparatus, computer equipment and storage medium, which obtain a preset vehicle image in a preset visual positioning task, obtain a preset boundary box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and obtain a preset color set, wherein the color set includes multiple preset colors; obtain preset image features and preset text features corresponding to the preset vehicle damaged area; obtain probability values of each preset color output by a multimodal pre-trained model according to the preset image features, the preset text features, the color set and a predefined acquisition method; generate scores for each preset color according to the probability values of each preset color and a predefined score model, and select the preset color with the highest score as the predicted color; obtain a loss value between the predicted color and the actual color of the preset vehicle damaged area marked, and train the multimodal pre-trained model with the goal of reducing the loss value. Training model; obtaining a current vehicle image in a current visual positioning task, obtaining a current color output by the trained multimodal pre-trained model, marking a damaged area of the current vehicle on the current vehicle image using the current color, and generating a marked current vehicle image. The beneficial effects are in two aspects. On the one hand, obtaining a current vehicle image in a current visual positioning task, obtaining a current color output by the trained multimodal pre-trained model, marking a damaged area of the current vehicle on the current vehicle image using the current color, and generating a marked current vehicle image. Since no manual marking is required, the marking time of the damaged area of the current vehicle in the current vehicle image is reduced, which is beneficial to improving the marking efficiency of the damaged area of the current vehicle in the current vehicle image. On the other hand, since the multimodal pre-trained model will not be affected by subjective factors, it is beneficial to improve the stability of the marked current vehicle image. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0040] Figure 1 is a schematic diagram of an application environment of a visual positioning method in one embodiment of the present invention;
[0041] Figure 2 A schematic diagram of a flow chart of a visual positioning method provided by an embodiment of the present invention;
[0042] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S23;
[0043] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S25;
[0044] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S26;
[0045] Figure 6 is a structural schematic diagram of a visual positioning device in one embodiment of the present invention;
[0046] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;
[0047] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an application environment of a visual positioning method according to an embodiment of the present invention. The visual positioning method provided by the embodiment of the present invention can be applied in the following embodiments: Figure 1 In an application environment, a client communicates with a server through a network.
[0050] The server obtains a preset vehicle image in a preset visual positioning task through the client, obtains a preset boundary box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and obtains a preset color set, wherein the color set includes a plurality of preset colors;
[0051] Acquire preset image features and preset text features corresponding to the preset damaged area of the vehicle;
[0052] According to the preset image feature, the preset text feature, the color set and a predefined acquisition method, obtaining a probability value of each of the preset colors output by the multimodal pre-training model;
[0053] Generate scores for the preset colors according to the probability values of the preset colors and a predefined score model, and select the preset color with the highest score as the predicted color;
[0054] Obtaining a loss value between the predicted color and an actual color marking the preset damaged area of the vehicle, training the multimodal pre-training model with the goal of reducing the loss value;
[0055] A current vehicle image in a current visual positioning task is obtained, a current color output by the trained multimodal pre-trained model is obtained, and on the current vehicle image, a damaged area of the current vehicle is marked with the current color to generate a marked current vehicle image.
[0056] In the scheme implemented by the above-mentioned visual positioning method, device, equipment and medium, the beneficial effects of the scheme are in two aspects. On the one hand, the current vehicle image in the current visual positioning task is obtained, and the current color output by the multimodal pre-trained model that has been trained is obtained. On the current vehicle image, the damaged area of the current vehicle is marked with the current color to generate the marked current vehicle image. Since no manual marking is required, the marking time of the damaged area of the current vehicle in the current vehicle image is reduced, which is conducive to improving the marking efficiency of the damaged area of the current vehicle in the current vehicle image. On the other hand, since the multimodal pre-trained model will not be affected by subjective factors, it is conducive to improving the stability of the marked current vehicle image.
[0057] The client may include but is not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.
[0058] The server can be implemented with an independent task database or a task database cluster composed of multiple task databases. The present invention is described in detail below through specific embodiments.
[0059] See also Figure 2 , Figure 2 A flow chart of a visual positioning method provided by an embodiment of the present invention includes the following steps:
[0060] S21, obtaining a preset vehicle image in a preset visual positioning task, obtaining a preset boundary box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and obtaining a preset color set, wherein the color set includes a plurality of preset colors;
[0061] Exemplarily, the color set includes a plurality of preset colors, and the plurality of preset colors are red, yellow, and green, respectively.
[0062] Among them, different color markings can be used to quickly distinguish different damaged areas of the vehicle.
[0063] For example, red is used to mark areas of severe damage to the vehicle, yellow is used to mark areas of moderate damage to the vehicle, and green is used to mark areas of minor damage to the vehicle.
[0064] For the sake of illustration, an example is given below:
[0065] For example, the preset vehicle damaged area is a vehicle seriously damaged area, and the actual color marking the preset vehicle damaged area is red.
[0066] For example, the preset vehicle damaged area is a vehicle medium damaged area, and the actual color marking the preset vehicle damaged area is yellow.
[0067] For example, the preset vehicle damaged area is an area where the vehicle is slightly damaged, and the actual color marking the preset vehicle damaged area is green.
[0068] S22, obtaining preset image features and preset text features corresponding to the preset damaged area of the vehicle;
[0069] Wherein, the obtaining of preset image features and preset text features corresponding to the preset damaged area of the vehicle includes:
[0070] Performing feature extraction on the preset bounding box to obtain preset image features corresponding to the preset damaged area of the vehicle;
[0071] Feature extraction is performed on the preset text prompt information to obtain preset text features corresponding to the preset vehicle damaged area.
[0072] S23, acquiring a probability value of each of the preset colors output by the multimodal pre-trained model according to the preset image feature, the preset text feature, the color set, and a predefined acquisition method;
[0073] S24, generating scores for the preset colors according to the probability values of the preset colors and a predefined score model, and selecting the preset color with the highest score as the predicted color;
[0074] The step of generating scores for each preset color according to the probability values of each preset color and a predefined score model, and selecting the preset color with the highest score as the predicted color comprises:
[0075] Obtaining a preset storage path, and importing the probability value of each preset color through the storage path;
[0076] According to the probability values of the respective preset colors and a predefined score model, scores of the respective preset colors are generated, and the preset color with the highest score is selected as the predicted color.
[0077] Exemplarily, the score model is:
[0078]
[0079] Among them, P j is the score of the jth preset color, c j is the probability of the jth preset color, c is the color set, and j is the serial number of the preset color.
[0080] For the sake of illustration, an example is given below:
[0081] For example, the probability of the first preset color is 0.5, the probability of the second preset color is 0.3, and the probability of the third preset color is 0.2. According to the score model, the score of the first preset color is 0.5, the score of the second preset color is 0.3, and the score of the third preset color is 0.2, so the first preset color is used as the predicted color.
[0082] For example, the probability of the first preset color is 0.3, the probability of the second preset color is 0.4, and the probability of the third preset color is 0.3. According to the score model, the score of the first preset color is 0.3, the score of the second preset color is 0.4, and the score of the third preset color is 0.3, so the second preset color is used as the predicted color.
[0083] For example, the probability of the first preset color is 0.3, the probability of the second preset color is 0.3, and the probability of the third preset color is 0.4. According to the score model, the score of the first preset color is 0.3, the score of the second preset color is 0.3, and the score of the third preset color is 0.4, so the third preset color is used as the predicted color.
[0084] S25, obtaining a loss value between the predicted color and the actual color of the preset vehicle damaged area, with the goal of reducing the loss value, and training the multimodal pre-training model;
[0085] S26, obtaining a current vehicle image in a current visual positioning task, obtaining a current color output by the multimodal pre-trained model that has been trained, marking a damaged area of the current vehicle on the current vehicle image using the current color, and generating a marked current vehicle image.
[0086] For the sake of illustration, an example is given below:
[0087] For example, the damaged area of the current vehicle is a severely damaged area of the vehicle, and the damaged area of the current vehicle is marked in red on the current vehicle image to generate the marked current vehicle image.
[0088] For example, if the damaged area of the current vehicle is a medium damaged area of the vehicle, the damaged area of the current vehicle is marked in yellow on the current vehicle image to generate the marked current vehicle image.
[0089] For example, the damaged area of the current vehicle is a slightly damaged area of the vehicle. On the current vehicle image, the actual color of the damaged area of the current vehicle is marked with green to generate the marked current vehicle image.
[0090] Among them, the beneficial effects of S26 are as follows: color marking can reduce misjudgment caused by human factors. At a complex accident scene, the damaged vehicle may have multiple damages. The use of color marking can clearly indicate the specific location and extent of each damage, avoiding omissions or confusion during the handling process.
[0091] During the accident handling process, color coding can serve as evidence of vehicle damage, helping accident handling personnel quickly understand the overall condition of the vehicle, thereby helping to speed up the progress of insurance claims.
[0092] Wherein, in the step of obtaining the current vehicle image in the current visual positioning task, obtaining the current color output by the multimodal pre-trained model after training, marking the damaged area of the current vehicle on the current vehicle image with the current color, and generating the marked current vehicle image, the visual positioning method includes:
[0093] Step A: connecting to a preset insurance system and sending the marked current vehicle image to the insurance system.
[0094] Exemplarily, in the step of obtaining a current vehicle image in the current visual positioning task, obtaining a current color output by the trained multimodal pre-trained model, marking a damaged area of the current vehicle on the current vehicle image using the current color, and generating the marked current vehicle image, the visual positioning method includes:
[0095] Step B, connecting the screen and displaying the marked current vehicle image on the screen.
[0096] Among them, step A is executed before or after step B. Step A can also be executed at the same time as step B. The specific execution order is not limited here.
[0097] In the embodiment of the present invention, the beneficial effects of the scheme are in two aspects. On the one hand, a current vehicle image in a current visual positioning task is obtained, and a current color output by the multimodal pre-trained model that has been trained is obtained. On the current vehicle image, the damaged area of the current vehicle is marked with the current color to generate the marked current vehicle image. Since no manual marking is required, the marking time of the damaged area of the current vehicle in the current vehicle image is reduced, which is beneficial to improving the marking efficiency of the damaged area of the current vehicle in the current vehicle image. On the other hand, since the multimodal pre-trained model will not be affected by subjective factors, it is beneficial to improve the stability of the marked current vehicle image.
[0098] See also Figure 3 , Figure 3 yes Figure 1 A specific implementation flow diagram of step S23 is described in detail as follows:
[0099] S31, based on a cross-modal attention mechanism, inputting the preset image features, the preset text features, and the color set into a multimodal pre-training model;
[0100] S32, obtaining the probability value of each of the preset colors output by the multimodal pre-training model based on the preset image features, the preset text features and the color set.
[0101] In an embodiment of the present invention, the probability value of each of the preset colors output by the multimodal pre-trained model based on the preset image features, the preset text features and the color set is obtained, so that the probability value of each of the preset colors can be obtained at one time.
[0102] See also Figure 4 , Figure 4 yes Figure 1 A specific implementation flow diagram of step S25 is described in detail as follows:
[0103] S41, obtaining a loss function, wherein the loss function includes one of a cross entropy loss function and a mean square error loss function;
[0104] S42, obtaining the loss value between the predicted color and the actual color marking the preset damaged area of the vehicle through the cross entropy loss function or the mean square error loss function, and training the multimodal pre-training model with the goal of reducing the loss value.
[0105] In an embodiment of the present invention, the multimodal pre-training model is trained with the goal of reducing the loss value, which can reduce the difference between the predicted color and the actual color of the preset vehicle damaged area, thereby improving the accuracy of the multimodal pre-training model.
[0106] See also Figure 5 , Figure 5 yes Figure 1 A specific implementation flow diagram of step S26 is described in detail as follows:
[0107] S51, during the training process of the multimodal pre-training model, when the loss value is less than a preset value, stop training the multimodal pre-training model and save the multimodal pre-training model after training;
[0108] S52, obtaining a current vehicle image in a current visual positioning task, obtaining a current vehicle damaged area of the current vehicle image, inputting the current vehicle damaged area into the trained multimodal pre-trained model, obtaining a current color output by the trained multimodal pre-trained model based on the current vehicle damaged area, marking the current vehicle damaged area on the current vehicle image with the current color, and generating a marked current vehicle image.
[0109] In the embodiment of the present invention, since manual marking is not required, the marking time of the damaged area of the current vehicle in the current vehicle image is reduced, which is conducive to improving the marking efficiency of the damaged area of the current vehicle in the current vehicle image.
[0110] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of a visual positioning device in one embodiment of the present invention. Figure 6 As shown, the visual positioning device includes a first acquisition module 101, a second acquisition module 102, a determination module 103, a selection module 104, a training module 105, and a generation module 106. The functional modules are described in detail as follows:
[0111] The first acquisition module 101 is used to acquire a preset vehicle image in a preset visual positioning task, acquire a preset boundary box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and acquire a preset color set, wherein the color set includes a plurality of preset colors;
[0112] The second acquisition module 102 is used to acquire the preset image features and preset text features corresponding to the preset damaged area of the vehicle;
[0113] A determination module 103, configured to obtain a probability value of each of the preset colors output by the multimodal pre-training model according to the preset image feature, the preset text feature, the color set, and a predefined acquisition method;
[0114] A selection module 104, configured to generate scores for each preset color according to the probability values of each preset color and a predefined score model, and select the preset color with the highest score as the predicted color;
[0115] A training module 105 is used to obtain a loss value between the predicted color and the actual color of the preset vehicle damaged area, and train the multimodal pre-training model with the goal of reducing the loss value;
[0116] The generation module 106 is used to obtain the current vehicle image in the current visual positioning task, obtain the current color output by the multimodal pre-trained model that has been trained, mark the damaged area of the current vehicle on the current vehicle image with the current color, and generate the marked current vehicle image.
[0117] In one embodiment, the second acquisition module 102 includes:
[0118] A first extraction subunit is used to perform feature extraction on the preset bounding box to obtain preset image features corresponding to the preset damaged area of the vehicle;
[0119] The second extraction subunit is used to perform feature extraction on the preset text prompt information to obtain preset text features corresponding to the preset vehicle damaged area.
[0120] In one embodiment, the determining module 103 includes:
[0121] An input subunit, used for inputting the preset image features, the preset text features, and the color set into a multimodal pre-training model based on a cross-modal attention mechanism;
[0122] An output subunit is used to obtain the probability value of each preset color output by the multimodal pre-training model based on the preset image feature, the preset text feature and the color set.
[0123] In one embodiment, the selection module 104 includes:
[0124] An import subunit, used for obtaining a preset storage path, and importing the probability value of each preset color through the storage path;
[0125] The selection subunit is used to generate scores for each preset color according to the probability values of each preset color and a predefined score model, and select the preset color with the highest score as the predicted color.
[0126] In one embodiment, the training module 105 includes:
[0127] An acquisition subunit, used to acquire a loss function, wherein the loss function includes one of a cross entropy loss function and a mean square error loss function;
[0128] The training subunit is used to obtain the loss value between the predicted color and the actual color of the preset vehicle damaged area marked by the cross entropy loss function or the mean square error loss function, so as to train the multimodal pre-training model with the goal of reducing the loss value.
[0129] In one embodiment, the generating module 106 includes:
[0130] A saving subunit, used for stopping the training of the multimodal pre-training model when the loss value is less than a preset value during the training process of the multimodal pre-training model, and saving the multimodal pre-training model after training;
[0131] A generating subunit is used to obtain a current vehicle image in a current visual positioning task, obtain a current vehicle damaged area of the current vehicle image, input the current vehicle damaged area into the trained multimodal pre-trained model, obtain a current color output by the trained multimodal pre-trained model based on the current vehicle damaged area, mark the current vehicle damaged area on the current vehicle image with the current color, and generate the marked current vehicle image.
[0132] In one embodiment, the visual positioning device further includes:
[0133] The display module is used to connect to a preset insurance system and send the marked current vehicle image to the insurance system.
[0134] In the embodiment of the present invention, the beneficial effects of the scheme are in two aspects. On the one hand, a current vehicle image in a current visual positioning task is obtained, and a current color output by the multimodal pre-trained model that has been trained is obtained. On the current vehicle image, the damaged area of the current vehicle is marked with the current color to generate the marked current vehicle image. Since no manual marking is required, the marking time of the damaged area of the current vehicle in the current vehicle image is reduced, which is beneficial to improving the marking efficiency of the damaged area of the current vehicle in the current vehicle image. On the other hand, since the multimodal pre-trained model will not be affected by subjective factors, it is beneficial to improve the stability of the marked current vehicle image.
[0135] For the specific limitations of the visual positioning device, please refer to the limitations of the visual positioning method above, which will not be repeated here.
[0136] Each module in the above-mentioned visual positioning device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0137] See also Figure 7 , Figure 7 is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a server. The internal structure diagram thereof may be as shown in FIG. Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus.
[0138] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, a function or step of a visual positioning method is implemented.
[0139] See also Figure 8 , Figure 8 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external task database through a network connection. When the computer program is executed by the processor, it implements the functions or steps of a visual positioning method.
[0140] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:
[0141] Obtain a preset vehicle image in a preset visual positioning task, obtain a preset bounding box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and obtain a preset color set, wherein the color set includes a plurality of preset colors;
[0142] Acquire preset image features and preset text features corresponding to the preset damaged area of the vehicle;
[0143] According to the preset image feature, the preset text feature, the color set and a predefined acquisition method, obtaining a probability value of each of the preset colors output by the multimodal pre-training model;
[0144] Generate scores for the preset colors according to the probability values of the preset colors and a predefined score model, and select the preset color with the highest score as the predicted color;
[0145] Obtaining a loss value between the predicted color and an actual color marking the preset damaged area of the vehicle, training the multimodal pre-training model with the goal of reducing the loss value;
[0146] A current vehicle image in a current visual positioning task is obtained, a current color output by the trained multimodal pre-trained model is obtained, and on the current vehicle image, a damaged area of the current vehicle is marked with the current color to generate a marked current vehicle image.
[0147] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.
[0148] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0149] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.
[0150] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0151] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.
[0152] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.
Claims
1. A visual positioning method, characterized in that: include: Obtain a preset vehicle image in a preset visual positioning task, obtain a preset boundary box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and obtain a preset color set, wherein the color set includes a plurality of preset colors; Acquire preset image features and preset text features corresponding to the preset damaged area of the vehicle; According to the preset image feature, the preset text feature, the color set and a predefined acquisition method, obtaining a probability value of each of the preset colors output by the multimodal pre-training model; Generate scores for the preset colors according to the probability values of the preset colors and a predefined score model, and select the preset color with the highest score as the predicted color; Obtaining a loss value between the predicted color and an actual color marking the preset damaged area of the vehicle, training the multimodal pre-training model with the goal of reducing the loss value; A current vehicle image in a current visual positioning task is obtained, a current color output by the trained multimodal pre-trained model is obtained, and on the current vehicle image, a damaged area of the current vehicle is marked with the current color to generate a marked current vehicle image.
2. The visual positioning method according to claim 1, characterized in that: The obtaining of preset image features and preset text features corresponding to the preset damaged area of the vehicle includes: Performing feature extraction on the preset bounding box to obtain preset image features corresponding to the preset damaged area of the vehicle; Feature extraction is performed on the preset text prompt information to obtain preset text features corresponding to the preset vehicle damaged area.
3. The visual positioning method according to claim 1, characterized in that: The step of obtaining the probability value of each of the preset colors output by the multimodal pre-trained model according to the preset image feature, the preset text feature, the color set, and a predefined acquisition method includes: Based on a cross-modal attention mechanism, the preset image features, the preset text features, and the color set are input into a multimodal pre-training model; Obtain probability values of each of the preset colors output by the multimodal pre-trained model based on the preset image features, the preset text features, and the color set.
4. The visual positioning method according to claim 1, characterized in that: Generating scores for each preset color according to the probability values of each preset color and a predefined score model, and selecting the preset color with the highest score as the predicted color, comprises: Obtaining a preset storage path, and importing the probability value of each preset color through the storage path; According to the probability values of the respective preset colors and a predefined score model, scores of the respective preset colors are generated, and the preset color with the highest score is selected as the predicted color.
5. The visual positioning method according to claim 1, characterized in that: The obtaining of the loss value between the predicted color and the actual color of the preset vehicle damaged area marked, with the goal of reducing the loss value, training the multimodal pre-training model, comprises: Obtaining a loss function, wherein the loss function includes one of a cross entropy loss function and a mean square error loss function; The loss value between the predicted color and the actual color of the preset vehicle damaged area marked is obtained through the cross entropy loss function or the mean square error loss function, and the multimodal pre-training model is trained with the goal of reducing the loss value.
6. The visual positioning method according to claim 1, characterized in that: The method of obtaining a current vehicle image in a current visual positioning task, obtaining a current color output by the multimodal pre-trained model after training, marking a damaged area of the current vehicle on the current vehicle image with the current color, and generating a marked current vehicle image includes: During the training process of the multimodal pre-training model, when the loss value is less than a preset value, the training of the multimodal pre-training model is stopped, and the trained multimodal pre-training model is saved; A current vehicle image in a current visual positioning task is obtained, a current vehicle damaged region of the current vehicle image is obtained, the current vehicle damaged region is input into the trained multimodal pre-trained model, a current color output by the trained multimodal pre-trained model based on the current vehicle damaged region is obtained, and on the current vehicle image, the current vehicle damaged region is marked with the current color to generate the marked current vehicle image.
7. The visual positioning method according to claim 1, characterized in that: In the step of obtaining the current vehicle image in the current visual positioning task, obtaining the current color output by the trained multimodal pre-trained model, marking the damaged area of the current vehicle on the current vehicle image with the current color, and generating the marked current vehicle image, the visual positioning method includes: A preset insurance system is connected, and the marked current vehicle image is sent to the insurance system.
8. A visual positioning device, characterized in that: include: A first acquisition module is used to acquire a preset vehicle image in a preset visual positioning task, acquire a preset boundary box and preset text prompt information corresponding to a preset vehicle damaged area in the preset vehicle image, and acquire a preset color set, wherein the color set includes a plurality of preset colors; A second acquisition module is used to acquire preset image features and preset text features corresponding to the preset damaged area of the vehicle; A determination module, configured to obtain a probability value of each of the preset colors output by the multimodal pre-trained model according to the preset image feature, the preset text feature, the color set, and a predefined acquisition method; A selection module, used to generate scores for each preset color according to the probability values of each preset color and a predefined score model, and select the preset color with the highest score as the predicted color; A training module, used for obtaining a loss value between the predicted color and an actual color marking the preset damaged area of the vehicle, and training the multimodal pre-training model with the goal of reducing the loss value; A generation module is used to obtain a current vehicle image in a current visual positioning task, obtain a current color output by the multimodal pre-trained model that has been trained, mark a damaged area of the current vehicle on the current vehicle image with the current color, and generate a marked current vehicle image.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the visual positioning method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the visual positioning method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Vehicle loss assessment method and device based on deep learning model, equipment and medium
CN111666995A
Damage diagram creation assistance device, damage diagram creation assistance method, damage diagram creation assistance program, and damage diagram creation assistance system
CN113196041A
Vehicle damage judgment system based on computer vision
CN114926618A
Heat map of vehicle damage
US9886771B1
