Control method and system for explainable monitoring and remote takeover based on vla decisions

Abstract

The application relates to the technical field of automatic driving safety monitoring, in particular to a control method and system for explainable monitoring and remote takeover based on VLA decision, wherein when a VLA model generates a decision, semantic information of the current decision and semantic information of a target region with the highest weight related to the current decision are extracted, and a semantic explanation of the current decision is constructed based on the semantic information of the current decision and the semantic information of the target region, so that the problem that an existing VLE model lacks explainability is solved. In addition, a rule library is constructed in advance, the semantic explanation is matched with the rule library, it is judged whether there is a conflict with the traffic rules in the park, if there is a conflict, the conflict probability is quantified further, and the remote takeover mode is decided based on the conflict probability, so that the takeover speed is improved, and the small car can be prevented from continuously running in the case of violating the traffic rules in the park.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving safety monitoring technology, specifically to a control method and system based on VLA decision-making for interpretable monitoring and remote takeover. Background Technology

[0002] With the development of intelligent transportation and artificial intelligence technologies, autonomous vehicles are increasingly being used in closed parks (such as science parks, university campuses, logistics parks, tourist attractions, etc.).

[0003] Despite the continuous advancements in autonomous driving technology, many current systems still exhibit black-box characteristics. Therefore, they suffer from the following drawbacks: (1) Existing autonomous vehicles rely on VLA (Vision-Language-Action) models for control, meaning the decision-making process of the VLA model lacks transparency and interpretability (e.g., unexplained braking), leading to increased accident rates and impacting user trust. (2) When generating abnormal decisions, existing VLA models often rely on manual video analysis from remote monitoring centers for takeover decisions, resulting in slow response times. (3) Because autonomous vehicles operate in closed parks, existing VLA models often cannot adapt to the rules of closed parks, making it easy for autonomous vehicles to violate speed limits in logistics parks (e.g., speeding through pedestrian crossings). Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a control method and system for interpretable monitoring and remote takeover based on VLA decision-making, so as to solve the problems in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The present invention provides a control method for interpretable monitoring and remote takeover based on VLA decision-making, comprising the following steps:

[0007] Obtain the semantic information of the current decision output by the policy module of the VLA model, and obtain the semantic information of the key region and key region output by the cross-attention mechanism matrix of the last layer of the VLA model, wherein the key region is the region with the largest weight that is related to the current decision;

[0008] The semantic information of the current decision and the semantic information of the key region are mapped to a pre-constructed interpretive template to obtain the semantic interpretation of the current decision output by the policy module of the VLA model;

[0009] The semantic interpretation is matched with a pre-built rule base, and when the semantic interpretation conflicts with any rule in the rule base, the conflict is quantified based on the weight of the key region, the semantic interpretation, and the rule that conflicts with the semantic interpretation to obtain the conflict probability.

[0010] The conflict probability is compared with a set first probability threshold and a second probability threshold. When the conflict probability is greater than the first probability threshold, a remote voice prompt is given to the remote driver. When the conflict probability is greater than the second probability threshold, the remote driver control is forcibly switched. The first probability threshold is less than the second probability threshold.

[0011] In one embodiment of this application, the semantic information of the current decision and the semantic information of the key region are mapped to a pre-constructed interpretive template to obtain the semantic interpretation of the current decision output by the policy module of the VLA model, including:

[0012] The semantic information of the current decision is mapped to the result content box of the explanatory template, and the semantic information of the key region is mapped to the cause content box of the explanatory template, so as to obtain the semantic explanation of the current decision output by the policy module of the VLA model.

[0013] In one embodiment of this application, the process of constructing the rule base includes:

[0014] Obtain the text of multiple traffic rules within the closed park;

[0015] The traffic rule text is structurally organized to obtain a standard structured traffic rule text, wherein the standard structured traffic rule text includes a scene description part and an action description part;

[0016] Each standard-structured traffic rule text is segmented into words, resulting in words from the scene section and words from the action section;

[0017] The words in the scene section and the words in the action section are vectorized to obtain multiple word vectors; and based on a pre-built classification dictionary, the words corresponding to the multiple word vectors of the traffic rule text are tagged with part-of-speech tags to obtain scene words and action words, or scene words, action words and numerical ranges. The classification dictionary includes pre-stored word vectors of multiple scene words and word vectors of multiple action words, and the numerical ranges are used to quantify the scene words or the action words.

[0018] A standard rule vector can be constructed based on the word vectors of scene words and action words in each traffic rule text, or based on the word vectors of scene words, action words, and numerical ranges in each traffic rule text.

[0019] Each standard rule vector is expanded by approximate words and conflict words to obtain expanded rule vectors, and a rule base is constructed based on multiple expanded rule vectors.

[0020] In one embodiment of this application, each standard rule vector is expanded by approximate word expansion and conflict word expansion to obtain an expanded rule vector, including:

[0021] For the word vector of an action word in the standard rule vector of an action word with a numerical range quantization, calculate the similarity with the word vector of each action word in the classification dictionary, and take action words in the classification dictionary with a similarity less than a first similarity threshold as conflict candidate words; and take action words in the classification dictionary with a similarity greater than or equal to a second similarity threshold as approximate words, and construct conflict candidate words based on approximate words and value ranges outside the numerical range, wherein the second similarity threshold is greater than the first similarity threshold;

[0022] For the word vector of an action word in the standard rule vector of an action word that does not have a numerical range quantization, calculate the similarity with the word vector of each action word in the classification dictionary, and take the action words in the classification dictionary with a similarity less than the first similarity threshold as conflict candidate words.

[0023] For the word vector of a scene word in the standard rule vector, calculate the similarity with the word vector of each scene word in the classification dictionary, and take the scene word with a similarity greater than the second similarity as an approximate candidate word;

[0024] The conflicting candidate words and the similar candidate words are screened respectively to obtain conflicting words and similar words;

[0025] Each standard rule vector is expanded by using conflicting and similar words to obtain extended rule vectors.

[0026] In one embodiment of this application, matching the semantic interpretation with a pre-built rule base includes:

[0027] The semantic information of the key region in the semantic interpretation is matched with the scene words of each extended rule vector in the rule base, and the rule corresponding to the scene word with the highest matching degree is taken as the target rule. The scene word with the highest matching degree is the scene word with the largest number in the semantic information of the key region in the semantic interpretation.

[0028] The semantic information of the current decision in the semantic interpretation is matched with the word vector of the action word in the target rule. If the action matches, the semantic interpretation is determined to match the target rule. If the action does not match, the semantic information of the current decision in the semantic interpretation is matched with the conflicting word in the target rule. If the conflict matches, the semantic interpretation is determined to conflict with the target rule. If the action does not match, the semantic interpretation is determined to be unrelated to the target rule.

[0029] In one embodiment of this application, the conflict is quantified based on the weight of the key region, the semantic interpretation, and the rule that conflicts with the semantic interpretation to obtain the conflict probability, including:

[0030] Extract target action words from the semantic information of the current decision in the semantic interpretation, or extract target action words and their numerical values, and extract target scene words from the semantic information of key regions in the semantic interpretation. The target action words are words in the semantic information of the current decision in the semantic interpretation that are consistent with conflicting words in the target rules, and the target scene words are words in the semantic information of key regions in the semantic interpretation that are consistent with scene words or similar words in the target rules.

[0031] Calculate the first similarity between the word vector of the target scene word and the word vector of the scene word in the standard rule vector of the target rule; and calculate the second similarity between the word vector of the target action word and the word vector of the action word in the standard rule vector of the target rule, or calculate the second similarity between the word vector of the target action word and the value of the target action word and the word vector of the action word in the standard rule vector of the target rule.

[0032] The conflict probability is obtained by weighting and summing the weights of the key regions, the first similarity, and the second similarity.

[0033] In one embodiment of this application, it further includes:

[0034] The semantic interpretation of the current decision and the key area are sent to the remote driver.

[0035] This application also provides a control system for interpretable monitoring and remote takeover based on VLA decision-making, including:

[0036] The acquisition module is used to acquire the semantic information of the current decision output by the policy module of the VLA model, and to acquire the key region and semantic information of the key region output by the cross-attention mechanism matrix of the last layer of the VLA model, wherein the key region is the region with the largest weight related to the current decision.

[0037] The semantic interpretation module is used to map the semantic information of the current decision and the semantic information of the key region to a pre-constructed interpretive template to obtain the semantic interpretation of the current decision output by the policy module of the VLA model;

[0038] The rule matching module is used to match the semantic interpretation with a pre-built rule base, and when the semantic interpretation conflicts with any rule in the rule base, it quantifies the conflict based on the weight of the key region, the semantic interpretation, and the rule that conflicts with the semantic interpretation to obtain the conflict probability.

[0039] The takeover control module is used to compare the conflict probability with a set first probability threshold and a second probability threshold, and to provide remote voice prompts to the remote driver when the conflict probability is greater than the first probability threshold; and to forcibly switch to remote driver control when the conflict probability is greater than the second probability threshold, wherein the first probability threshold is less than the second probability threshold.

[0040] This application also provides an electronic device, including: a processor and a memory;

[0041] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to cause the electronic device to perform the methods described above.

[0042] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0043] The beneficial effects of this invention are as follows: The interpretable monitoring and remote takeover control method and system based on VLA decision-making of this invention extracts the semantic information of the current decision and the semantic information of the target region with the highest weight related to the current decision when the VLA model generates a decision. Based on the semantic information of the current decision and the semantic information of the target region, a semantic interpretation of the current decision is constructed. This solves the problem of the lack of interpretability in existing VLE models. Furthermore, this application pre-builds a rule base, matches the semantic interpretation with the rule base, and determines whether it conflicts with traffic rules within the park. If a conflict is found, the conflict probability is further quantified, and the remote takeover method is determined based on the conflict probability, thereby improving the takeover speed and preventing vehicles from continuously operating while violating traffic rules within the park. Attached Figure Description

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0045] Figure 1 This is a flowchart illustrating a control method for interpretable monitoring and remote takeover based on VLA decision-making in one embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating the rule base construction process in one embodiment of this application;

[0047] Figure 3This is a flowchart illustrating the takeover control process in one embodiment of this application;

[0048] Figure 4 This is a structural diagram of a control system for interpretable monitoring and remote takeover based on VLA decision-making, as shown in one embodiment of this application. Detailed Implementation

[0049] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0050] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the layers related to the present invention and are not drawn according to the actual number, shape and size of the layers in the actual implementation. In the actual implementation, the form, number and proportion of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0051] Numerous details are explored in the following description to provide a more thorough explanation of embodiments of the invention; however, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details.

[0052] Figure 1 This is a flowchart illustrating a control method for interpretable monitoring and remote takeover based on VLA decision-making, as shown in one embodiment of this application. Figure 1 As shown, the control method for interpretable monitoring and remote takeover based on VLA decision-making in this embodiment includes the following specific steps:

[0053] S110, obtain the semantic information of the current decision output by the policy module of the VLA model, and obtain the key region and semantic information of the key region output by the cross attention mechanism matrix of the last layer of the VLA model, wherein the key region is the region with the largest weight related to the current decision;

[0054] In this application, the policy module of the VLA model outputs decisions (such as "turn left" or "decelerate"), which are then converted into natural language descriptions (such as "the system decides to turn left to avoid the obstacle ahead"). A pre-trained language model (such as BERT) is used to encode the decision descriptions into semantic vectors.

[0055] The key region in the output of the last layer's cross-attention mechanism matrix of the VLA model is the region with the highest weights relevant to the current decision. The attention weight matrix of the last layer is extracted from the cross-attention module of the VLA model (such as the multimodal Transformer). ,in, The length of the input feature (e.g., the number of image regions). The number of queries related to decision-making;

[0056] Attention weight matrix Perform normalization and calculate the weight for each input region. And the region with the highest weight is designated as the key region.

[0057] S120, the semantic information of the current decision and the semantic information of the key region are mapped to a pre-constructed interpretive template to obtain the semantic interpretation of the current decision output by the policy module of the VLA model;

[0058] This application utilizes a constructed interpretive template to build semantic interpretations, the specific process of which includes:

[0059] The semantic information of the current decision is mapped to the result content box of the explanatory template, and the semantic information of the key region is mapped to the cause content box of the explanatory template, so as to obtain the semantic explanation of the current decision output by the policy module of the VLA model.

[0060] For example, the result is "braking", and the reason is "[visual area #23] detected pedestrian (confidence 92%)".

[0061] By pre-building templates, semantic explanations can be quickly constructed, providing causal explanations for the VLA model's decisions. The semantic explanation of the current decision, along with the key regions, is then sent to the remote driver to provide visual cues.

[0062] S130, the semantic interpretation is matched with a pre-built rule base, and when the semantic interpretation conflicts with any rule in the rule base, the conflict is quantified based on the weight of the key region, the semantic interpretation, and the rule that conflicts with the semantic interpretation to obtain the conflict probability;

[0063] To address the potential issue of autonomous vehicles violating park traffic rules, this application pre-builds a rule base. Figure 2 Here is a flowchart of the rule base construction process in one embodiment of this application, such as Figure 2 The process is as follows:

[0064] (1) Obtain multiple traffic rule texts within the closed park area;

[0065] Traffic rule texts are extracted from paper / electronic documents provided by the park management, traffic signs, and real-time rules captured by cameras (such as "No Parking" and "Speed ​​Limit 20km / h").

[0066] (2) The traffic rule text is structurally organized to obtain a standard structure traffic rule text, wherein the standard structure traffic rule text includes a scene description part and an action description part;

[0067] Then, the unstructured rule text is split into standard structures, for example:

[0068] [Scene Description] On the main road of the park; [Action Description] Drive on the right;

[0069] [Scene Description] On a special road section in XX; [Action Description] Making a U-turn in the rightmost lane;

[0070] [Scene Description] There is a pedestrian crossing ahead; [Action Description] The driving speed does not exceed 10km / h.

[0071] Structured processing can use BERT-based sequence labeling models (such as CRF, BiLSTM-CRF) to identify scenes and action entities. Alternatively, it can be done manually.

[0072] (3) Segment the text of each standard traffic rule to obtain words from the scene part and words from the action part;

[0073] Word segmentation can be performed using existing word segmentation tools, which will not be elaborated upon here.

[0074] (4) Vectorize the words in the scene part and the words in the action part to obtain multiple word vectors; and based on the pre-built classification dictionary, perform part-of-speech tagging on the words corresponding to the multiple word vectors of the traffic rule text to obtain scene words and action words, or obtain scene words, action words and numerical ranges, wherein the classification dictionary includes pre-stored word vectors of multiple scene words and word vectors of multiple action words, and the numerical range is used to quantify the scene words or to quantify the action words;

[0075] In this application, a pre-trained word2Vec model is used to convert words into word vectors.

[0076] Part-of-speech tagging relies on a pre-built classification dictionary. Since the vocabulary of scene words and action words related to traffic behavior is limited, a direct and accurate classification dictionary is chosen for part-of-speech tagging. This classification dictionary is a pre-built dictionary containing scene words (such as "main road" and "parking lot") and action words (such as "keep right" and "slow down").

[0077] During annotation, the cosine similarity between words with known parts of speech in the classification dictionary and word vectors is calculated. If the similarity is high, the part of speech is assigned to the corresponding word.

[0078] Since the scene description part and the action description part are pre-divided, the part-of-speech tagging rules stipulate that scene words come from the scene description part and action words come from the action description part.

[0079] For rules regarding the range of accessed values, such as "[Scene Description] There is a pedestrian crossing ahead; [Action Description] The driving speed does not exceed 10km / h", the range is converted into a numerical range. ".

[0080] (5) Construct standard rule vectors based on the word vectors of scene words and action words in each traffic rule text, or construct standard rule vectors based on the word vectors of scene words, action words, and numerical ranges in each traffic rule text.

[0081] Finally, by analyzing the word vectors of the scene words in each traffic rule text... Word vectors of action words and numerical range Combining them to form standard rule vectors ;

[0082]

[0083] For example, standard rule vectors It can be ;

[0084] (6) Perform approximate word expansion and conflict word expansion on each standard rule vector to obtain the expanded rule vector, and construct a rule base based on multiple expanded rule vectors.

[0085] Since the standard rule vectors in the rule base only contain a standard scene word, action word, and numerical range, it is difficult to directly infer whether the decision matches or conflicts with the rule when performing rule matching. In order to facilitate quick decision judgment, this application also performs approximation word expansion and conflict word expansion on each standard rule vector. The specific process is as follows:

[0086] (6-1) For the word vector of the action word in the standard rule vector of the action word with numerical range quantization, calculate the similarity with the word vector of each action word in the classification dictionary, and take the action words in the classification dictionary with similarity less than the first similarity threshold as conflict candidate words; and take the action words in the classification dictionary with similarity greater than or equal to the second similarity threshold as approximate words, and construct conflict candidate words based on approximate words and value ranges outside the numerical range, wherein the second similarity threshold is greater than the first similarity threshold;

[0087] There are two scenarios for conflicting actions. First, the verbs themselves are conflicting, such as the conflicting verb "decelerate" in "accelerate". Second, the verbs are similar but their ranges of meaning conflict, such as "speed limit". The conflicting word is "speed 20". Other cases where a conflict cannot be directly proven, such as "decelerate" and "turn left", do not require processing.

[0088] (6-2) For the word vector of the action word in the standard rule vector of the action word that does not have a numerical range quantization, calculate the similarity with the word vector of each action word in the classification dictionary, and take the action words in the classification dictionary with a similarity less than the first similarity threshold as conflict candidate words.

[0089] In this application, conflict candidate words are screened by calculating the cosine similarity between word vectors. The cosine similarity is between [-1, 1]. If the similarity is less than 0, it indicates that the word may be a conflict word.

[0090] (6-3) For the word vector of the scene word in the standard rule vector, calculate the similarity with the word vector of each scene word in the classification dictionary, and take the scene word with a similarity greater than the second similarity as the approximate candidate word;

[0091] In addition to addressing conflicts, this application also expands the word vectors of scene words in the standard rule vectors with approximate words to facilitate rapid matching of similar semantics. Words similar to the scene words are added to the rule vectors.

[0092] (6-4) The conflicting candidate words and the similar candidate words are screened respectively to obtain conflicting words and similar words;

[0093] (6-5) Based on the conflicting and similar words of each standard rule vector, each standard rule vector is expanded to obtain an extended rule vector.

[0094] Finally, through manual screening, conflicting and similar words that meet the semantic requirements are identified and expanded into the standard rule vector to obtain the extended rule vector, for example:

[0095]

[0096] Similar words to those indicating a scene. Conflicting words that indicate action words.

[0097] After building the rule base, the semantic interpretation of the current decision can be quickly matched with the rule base. The matching process includes:

[0098] S1301, Match the semantic information of the key region in the semantic interpretation with the scene words of each extended rule vector in the rule base, and take the rule corresponding to the scene word with the highest matching degree as the target rule, wherein the scene word with the highest matching degree is the scene word with the largest number in the semantic information of the key region in the semantic interpretation;

[0099] When performing a match, the scene is first determined, and the scene information in the semantic interpretation is contained in the semantic information of the key regions.

[0100] For example:

[0101] Semantic information of key regions: [Visual region #23] detects pedestrians (confidence 92%);

[0102] The scenario word for rule N in the rule base is: passerby, and similar words include: personnel, pedestrian, crowd, etc.

[0103] During matching, the semantic information of "pedestrian" in the key region is compared with scene words and similar words. If a matching word vector exists, the corresponding rule is adopted as the target rule. If multiple scene words or similar words appear in the semantic information of the key region, the one with the highest frequency is adopted as the target rule. If the frequency is the same, the target rule is selected according to the similarity (similar words are pre-sorted according to similarity, with higher-ranking similar words having higher similarity).

[0104] S1302, perform action matching between the semantic information of the current decision in the semantic interpretation and the word vector of the action word in the target rule. When the action matches, determine that the semantic interpretation matches the target rule. When the action does not match, perform conflict matching between the semantic information of the current decision in the semantic interpretation and the conflicting word in the target rule. When the conflict matches, determine that the semantic interpretation conflicts with the target rule. When the conflict does not match, determine that the semantic interpretation is not related to the target rule.

[0105] The principle of action matching is the same as that of scene matching, except that numerical matching is added. The specific process will not be elaborated here, but can be understood by referring to the following examples.

[0106] For example, the semantic interpretation is: the result is "acceleration", and the reason is "sidewalk, no obstacles ahead";

[0107] Rule N is: Scene words: words similar to "zebra crossing" {pedestrian walkway, crosswalk}, action words conflicting with "decelerate" {accelerate, speed up};

[0108] First, the scene is matched. In the semantic interpretation, "sidewalk" is consistent with the similar word "sidewalk" in the scene word in rule N. Therefore, rule N is selected as the target rule.

[0109] Next, action matching is performed. "Accelerate" does not match the action word "decelerate" in rule N. Therefore, it is determined that the current decision does not conform to the content described in the rule. Therefore, further conflict matching is performed. "Accelerate" matches the conflict word "accelerate" in the action word in rule N. Therefore, it is determined that there may be a conflict.

[0110] If a rule neither matches nor conflicts with another rule (e.g., "accelerate" and "turn right"), then that target rule is discarded, and the next target rule is matched. If none of the target rules match or conflict, then no action is taken.

[0111] In the above process, since approximate word matching is used for fast matching, the semantic accuracy of approximate word matching is not high. If a potentially conflicting rule has been matched, the conflict needs to be further quantified based on the weight of the key region, the semantic interpretation, and the rule that conflicts with the semantic interpretation to obtain an accurate conflict probability. The quantified conflict probability is then used to accurately determine whether a conflict exists. Specifically, this includes:

[0112] S1311, extract target action words from the semantic information of the current decision in the semantic interpretation, or extract target action words and the numerical values ​​of the target action words, and extract target scene words from the semantic information of the key regions in the semantic interpretation, wherein the target action words are words in the semantic information of the current decision in the semantic interpretation that are consistent with conflicting words in the target rules, and the target scene words are words in the semantic information of the key regions in the semantic interpretation that are consistent with scene words or similar words in the target rules;

[0113] The target action word is the action word in the target rule obtained through matching mentioned above. Conflicting words The target scenario words are the scenario words in the target rules. Or scene words Similar words .

[0114] As mentioned earlier, in some cases the target action word has a numerical value, such as "speed 20".

[0115] S1312, calculate the first similarity between the word vector of the target scene word and the word vector of the scene word in the standard rule vector of the target rule; and calculate the second similarity between the word vector of the target action word and the word vector of the action word in the standard rule vector of the target rule, or calculate the second similarity between the word vector of the target action word and the value of the target action word and the word vector of the action word in the standard rule vector of the target rule.

[0116] The scene words in the standard rule vector of the target rule and action words As a standard semantic expression, the overall conflict probability is constructed by calculating the similarity between the matching word and the standard word.

[0117] For standard rule vectors that do not have a numerical range, the first similarity... Second similarity The mathematical expressions are as follows:

[0118]

[0119]

[0120] For standard rule vectors that have a numerical range, it is not necessary to calculate conflicting words. words related to actions The similarity between them needs to be calculated using action words. words related to actions similarity If similarity If the similarity is less than the preset similarity threshold, no comparison is needed. The numerical range and If the value is zero, the similarity score is set to 0.

[0121] If similarity If the similarity is greater than or equal to the preset similarity threshold, then further comparison is performed. The numerical range and The value, The value did not fall into When the value is within the specified range, the similarity is directly set to 0.

[0122] If similarity Greater than or equal to the preset similarity threshold, and The value falls into When the numerical range is within a certain range, the similarity will be... As the second similarity.

[0123] in,

[0124]

[0125] S1313, the weights of the key regions, the first similarity, and the second similarity are weighted and summed to obtain the conflict probability.

[0126] Finally, the weights of key areas. The first similarity and the second similarity We perform a weighted summation to obtain the conflict probability. The mathematical expression is:

[0127]

[0128] In the formula, As the first weight, As the second weight, It is the third weight.

[0129] For example, the first weight Second weight Third weight The values ​​can be 0.4, 0.3, and 0.3, respectively, and are empirical values.

[0130] S140, compare the conflict probability with a set first probability threshold and a second probability threshold, and when the conflict probability is greater than the first probability threshold, provide a remote voice prompt to the remote driver; when the conflict probability is greater than the second probability threshold, forcibly switch to remote driver control, wherein the first probability threshold is less than the second probability threshold.

[0131] Figure 3 This is a flowchart of the takeover control process in one embodiment of this application, such as... Figure 3 As shown, the final remote control handover is performed based on the conflict probability, including:

[0132] L1 Warning: The voice prompts the driver.

[0133] L2 forced takeover: Switch to remote cockpit control.

[0134] Remote takeover, achieved through rapid matching of semantic information from the rule base, has a faster response time and enables the generation of action semantic interpretations (latency ≤ 50ms); rule conflict detection accuracy ≥ 99%; and emergency takeover trigger time ≤ 100ms.

[0135] This invention discloses a control method for interpretable monitoring and remote takeover based on VLA decision-making. When the VLA model generates a decision, this application extracts the semantic information of the current decision and the semantic information of the target region with the highest weight related to the current decision. Based on the semantic information of the current decision and the target region, a semantic interpretation of the current decision is constructed. This solves the problem of the lack of interpretability in existing VLE models. Furthermore, this application pre-builds a rule base, matches the semantic interpretation with the rule base, and determines whether it conflicts with traffic rules within the park. If a conflict is found, the conflict probability is further quantified, and the remote takeover method is determined based on the conflict probability, thereby improving the takeover speed and preventing vehicles from continuously operating while violating traffic rules within the park.

[0136] like Figure 4 As shown, this application also provides a control system for interpretable monitoring and remote takeover based on VLA decision-making, including:

[0137] The acquisition module is used to acquire the semantic information of the current decision output by the policy module of the VLA model, and to acquire the key region and semantic information of the key region output by the cross-attention mechanism matrix of the last layer of the VLA model, wherein the key region is the region with the largest weight related to the current decision.

[0138] The semantic interpretation module is used to map the semantic information of the current decision and the semantic information of the key region to a pre-constructed interpretive template to obtain the semantic interpretation of the current decision output by the policy module of the VLA model;

[0139] The rule matching module is used to match the semantic interpretation with a pre-built rule base, and when the semantic interpretation conflicts with any rule in the rule base, it quantifies the conflict based on the weight of the key region, the semantic interpretation, and the rule that conflicts with the semantic interpretation to obtain the conflict probability.

[0140] The takeover control module is used to compare the conflict probability with a set first probability threshold and a second probability threshold, and to provide remote voice prompts to the remote driver when the conflict probability is greater than the first probability threshold; and to forcibly switch to remote driver control when the conflict probability is greater than the second probability threshold, wherein the first probability threshold is less than the second probability threshold.

[0141] This invention relates to a control system for interpretable monitoring and remote takeover based on VLA (Variable Access Decision) modeling. When generating decisions using the VLA model, this application extracts the semantic information of the current decision and the semantic information of the target region with the highest weight related to the current decision. Based on the semantic information of the current decision and the target region, a semantic interpretation of the current decision is constructed. This solves the problem of the lack of interpretability in existing VLA models. Furthermore, this application pre-builds a rule base, matches the semantic interpretation against the rule base, and determines whether it conflicts with traffic rules within the park. If a conflict is found, the conflict probability is further quantified, and the remote takeover method is determined based on the conflict probability, thereby improving the takeover speed and preventing vehicles from continuously operating while violating traffic rules within the park.

[0142] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0143] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory so that the terminal performs any of the methods in this embodiment.

[0144] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0145] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0146] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0147] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0148] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. The embodiments of the invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0149] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A control method for interpretable monitoring and remote takeover based on VLA decision-making, characterized in that, include: Obtain the semantic information of the current decision output by the policy module of the VLA model, and obtain the semantic information of the key region and key region output by the cross-attention mechanism matrix of the last layer of the VLA model, wherein the key region is the region with the largest weight that is related to the current decision; The semantic information of the current decision and the semantic information of the key region are mapped to a pre-constructed interpretive template to obtain the semantic interpretation of the current decision output by the policy module of the VLA model; The semantic interpretation is matched with a pre-built rule base, and when the semantic interpretation conflicts with any rule in the rule base, the conflict is quantified based on the weight of the key region, the semantic interpretation, and the rule that conflicts with the semantic interpretation to obtain the conflict probability. The conflict probability is compared with a set first probability threshold and a second probability threshold. When the conflict probability is greater than the first probability threshold, a remote voice prompt is given to the remote driver. When the conflict probability is greater than the second probability threshold, the remote driver control is forcibly switched. The first probability threshold is less than the second probability threshold.

2. The control method for interpretable monitoring and remote takeover based on VLA decision-making according to claim 1, characterized in that, The semantic information of the current decision and the semantic information of the key region are mapped to a pre-constructed interpretive template to obtain the semantic interpretation of the current decision output by the policy module of the VLA model, including: The semantic information of the current decision is mapped to the result content box of the explanatory template, and the semantic information of the key region is mapped to the cause content box of the explanatory template, so as to obtain the semantic explanation of the current decision output by the policy module of the VLA model.

3. The control method for interpretable monitoring and remote takeover based on VLA decision-making according to claim 1, characterized in that, The construction process of the rule base includes: Obtain the text of multiple traffic rules within the closed park; The traffic rule text is structurally organized to obtain a standard structured traffic rule text, wherein the standard structured traffic rule text includes a scene description part and an action description part; Each standard-structured traffic rule text is segmented into words to obtain words for the scene part and words for the action part; The words in the scene section and the words in the action section are vectorized to obtain multiple word vectors; and based on a pre-built classification dictionary, the words corresponding to the multiple word vectors of the traffic rule text are tagged with part-of-speech tags to obtain scene words and action words, or scene words, action words and numerical ranges. The classification dictionary includes pre-stored word vectors of multiple scene words and word vectors of multiple action words, and the numerical ranges are used to quantify the scene words or the action words. A standard rule vector can be constructed based on the word vectors of scene words and action words in each traffic rule text, or based on the word vectors of scene words, action words, and numerical ranges in each traffic rule text. Each standard rule vector is expanded by approximate words and conflict words to obtain expanded rule vectors, and a rule base is constructed based on multiple expanded rule vectors.

4. The control method for interpretable monitoring and remote takeover based on VLA decision-making according to claim 3, characterized in that, Each standard rule vector is expanded using approximation word expansion and conflict word expansion to obtain expanded rule vectors, including: For the word vector of an action word in the standard rule vector of an action word with a numerical range quantization, calculate the similarity with the word vector of each action word in the classification dictionary, and take action words in the classification dictionary with a similarity less than a first similarity threshold as conflict candidate words; and take action words in the classification dictionary with a similarity greater than or equal to a second similarity threshold as approximate words, and construct conflict candidate words based on approximate words and value ranges outside the numerical range, wherein the second similarity threshold is greater than the first similarity threshold; For the word vector of an action word in the standard rule vector of an action word that does not have a numerical range quantization, calculate the similarity with the word vector of each action word in the classification dictionary, and take the action words in the classification dictionary with a similarity less than the first similarity threshold as conflict candidate words. For the word vector of a scene word in the standard rule vector, calculate the similarity with the word vector of each scene word in the classification dictionary, and take the scene word with a similarity greater than the second similarity as an approximate candidate word; The conflicting candidate words and the similar candidate words are screened respectively to obtain conflicting words and similar words; Each standard rule vector is expanded by using conflicting and similar words to obtain extended rule vectors.

5. The control method for interpretable monitoring and remote takeover based on VLA decision-making according to claim 4, characterized in that, Matching the semantic interpretation with a pre-built rule base includes: The semantic information of the key region in the semantic interpretation is matched with the scene words of each extended rule vector in the rule base, and the rule corresponding to the scene word with the highest matching degree is taken as the target rule. The scene word with the highest matching degree is the scene word with the largest number in the semantic information of the key region in the semantic interpretation. The semantic information of the current decision in the semantic interpretation is matched with the word vector of the action word in the target rule. If the action matches, the semantic interpretation is determined to match the target rule. If the action does not match, the semantic information of the current decision in the semantic interpretation is matched with the conflicting word in the target rule. If the conflict matches, the semantic interpretation is determined to conflict with the target rule. If the action does not match, the semantic interpretation is determined to be unrelated to the target rule.

6. The control method for interpretable monitoring and remote takeover based on VLA decision-making according to claim 5, characterized in that, The conflict is quantified based on the weight of the key region, the semantic interpretation, and the rules that conflict with the semantic interpretation to obtain the conflict probability, including: Extract target action words from the semantic information of the current decision in the semantic interpretation, or extract target action words and their numerical values, and extract target scene words from the semantic information of key regions in the semantic interpretation. The target action words are words in the semantic information of the current decision in the semantic interpretation that are consistent with conflicting words in the target rules, and the target scene words are words in the semantic information of key regions in the semantic interpretation that are consistent with scene words or similar words in the target rules. Calculate the first similarity between the word vector of the target scene word and the word vector of the scene word in the standard rule vector of the target rule; and calculate the second similarity between the word vector of the target action word and the word vector of the action word in the standard rule vector of the target rule, or calculate the second similarity between the word vector of the target action word and the value of the target action word and the word vector of the action word in the standard rule vector of the target rule. The conflict probability is obtained by weighting and summing the weights of the key regions, the first similarity, and the second similarity.

7. The control method for interpretable monitoring and remote takeover based on VLA decision-making according to claim 1, characterized in that, Also includes: The semantic interpretation of the current decision and the key area are sent to the remote driver.

8. A control system for interpretable monitoring and remote takeover based on VLA decision-making, characterized in that, include: The acquisition module is used to acquire the semantic information of the current decision output by the policy module of the VLA model, and to acquire the key region and semantic information of the key region output by the cross-attention mechanism matrix of the last layer of the VLA model, wherein the key region is the region with the largest weight related to the current decision. The semantic interpretation module is used to map the semantic information of the current decision and the semantic information of the key region to a pre-constructed interpretive template to obtain the semantic interpretation of the current decision output by the policy module of the VLA model; The rule matching module is used to match the semantic interpretation with a pre-built rule base, and when the semantic interpretation conflicts with any rule in the rule base, it quantifies the conflict based on the weight of the key region, the semantic interpretation, and the rule that conflicts with the semantic interpretation to obtain the conflict probability. The takeover control module is used to compare the conflict probability with a set first probability threshold and a second probability threshold, and to provide remote voice prompts to the remote driver when the conflict probability is greater than the first probability threshold; and to forcibly switch to remote driver control when the conflict probability is greater than the second probability threshold, wherein the first probability threshold is less than the second probability threshold.

9. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images