A vehicle control method, device, mobile terminal, vehicle and storage medium

Abstract

The present application relates to a kind of vehicle control method, device, mobile terminal, vehicle and storage medium, mobile terminal is by obtaining vehicle real scene image;One or more interactive hotspots are generated in vehicle real scene image, wherein each interactive hotspot corresponds a vehicle controllable part in vehicle real scene image;In response to the first operation for interactive hotspot, show control car instruction panel, wherein control car instruction panel is the control car instruction panel of the vehicle controllable part corresponding to interactive hotspot;In response to the second operation for control car instruction panel, control vehicle.The present application converts control car entry from abstract menu to the real vehicle part that user shoots, realizes the control car experience of seen is obtained, reduces control car search cost, reduces misoperation, improves control car accuracy and safety.

Description

Technical Field

[0001] This invention relates to the field of vehicle control technology, specifically to a vehicle control method, device, mobile terminal, vehicle, and storage medium. Background Technology

[0002] With the rapid development of vehicle electronic and electrical architecture, vehicle-to-everything (V2X) technology, and mobile service applications, car owners can remotely control vehicle functions such as doors, windows, air conditioning, seats, trunk, charging ports, lights, and sunroofs via mobile applications. Existing vehicle control entry points are typically organized by function category, vehicle model page, or menu hierarchy. When there are many vehicle parts, complex function distribution, or when users are unfamiliar with the names of vehicle parts, users need to search for the target function in multiple menus. For example, when a user stands outside the car and sees a charging port cover, trunk, or a door, their actual intention is to directly control the part in front of them. However, existing applications still require users to understand the part name and find the corresponding entry in the function list, resulting in a long interaction path and the risk of misselection when multiple parts are displayed simultaneously. Summary of the Invention

[0003] To address at least one technical problem in the existing technology, the present invention adopts the following technical solution: In a first aspect, the present invention proposes a vehicle control method applied to a mobile terminal, the method comprising: Acquire real-view images of the vehicle; One or more interactive hotspots are generated in the real-world vehicle image, where each interactive hotspot corresponds to a controllable part of the vehicle in the real-world vehicle image; In response to the first operation on the interactive hotspot, the vehicle control command panel is displayed, which is the vehicle control command panel for the controllable part of the vehicle corresponding to the interactive hotspot. The vehicle is controlled in response to a second operation on the vehicle control command panel.

[0004] In some embodiments, after acquiring the real-view image of the vehicle, the method further includes: The quality inspection of real-world vehicle images includes multiple inspection items. If any detection item fails to meet the corresponding detection threshold, a prompt will be made to reacquire the real-world image of the vehicle.

[0005] In some embodiments, the plurality of detection items include: The system includes sharpness detection, exposure detection, occlusion detection, target proportion detection, and vehicle-related region detection. Among these, target proportion detection is used to detect the area of ​​a vehicle in a real-world vehicle image, while vehicle-related region detection is used to determine whether a vehicle exists in the real-world vehicle image.

[0006] In some embodiments, the method further includes: Identify controllable parts of a vehicle in a real-world image and obtain one or more identification results. Each identification result includes the part category of a controllable part and the location information of the controllable part.

[0007] In some embodiments, the controllable parts of a vehicle in a real-world vehicle image are identified to obtain one or more identification results, including: The real-world image of the vehicle is input into a pre-trained vehicle part recognition model for image processing, resulting in one or more recognition results output by the vehicle part recognition model.

[0008] In some embodiments, the position of the interactive hotspot in the real-world image of the vehicle is determined by the position information of the controllable part of the vehicle corresponding to the interactive hotspot.

[0009] In some embodiments, the location information of the controllable part of the vehicle includes: a detection frame or a segmentation mask; the interactive hotspot includes: an interactive origin and an interactive radius; The interactive origin can be any of the following: The center of the detection frame; The center of the segmented mask; Preset interaction priority points are determined in advance based on the category of controllable parts of the vehicle.

[0010] In some embodiments, the interactive radius is determined based on the area of ​​the controllable part of the vehicle, the screen size, and the distance between adjacent parts.

[0011] In some embodiments, if the distance between two adjacent interactive hotspots is less than a preset distance threshold, the positions of the two adjacent interactive hotspots are adjusted, or the local area where the two adjacent interactive hotspots are located is enlarged.

[0012] In some embodiments, the identification result further includes: controllable attributes, which are used to identify whether the controllable parts of the vehicle support remote control; Generate one or more interactive hotspots in the real-world image of the vehicle, including: For any recognition result, if the controllable attribute in the recognition result indicates that the controllable part of the vehicle supports remote control, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

[0013] In some embodiments, the identification result further includes: confidence level, which is used to identify the degree of trustworthiness of controllable parts of the vehicle; Generate one or more interactive hotspots in the real-world image of the vehicle, including: For any recognition result, if the confidence level in the recognition result is greater than or equal to the first confidence level threshold, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

[0014] In some embodiments, the method further includes: If the confidence level in the identification result is less than the first confidence threshold but greater than or equal to the second confidence threshold, then a hot spot to be confirmed corresponding to the controllable part of the vehicle is generated, wherein the first confidence threshold is greater than the second confidence threshold.

[0015] In some embodiments, the control commands in the vehicle control command panel are determined by the part type of the controllable part of the vehicle corresponding to the interactive hotspot.

[0016] In some embodiments, the recognition result further includes: scene identifier, which is used to identify whether the real-world image of the vehicle is taken from outside the vehicle or inside the vehicle; The method also includes: adjusting the control commands in the vehicle control command panel or adjusting the execution risk level of the control commands based on the scene identifier.

[0017] In some embodiments, in response to a first operation on an interactive hotspot, a vehicle control command panel is displayed, including: In response to the first action on the interactive hotspot, vehicle status information is obtained from the server; Based on vehicle status information, determine the executable commands for the controllable parts of the vehicle corresponding to the interactive hotspots; The vehicle control command panel is displayed, showing the executable commands.

[0018] In some embodiments, controlling the vehicle in response to a second operation on the vehicle control command panel includes: In response to the second operation on the vehicle control command panel, confirm the vehicle control command; Send vehicle control commands to the server; Receive the execution result of the vehicle control command sent by the server.

[0019] Secondly, the present invention provides a vehicle control device for use in a mobile terminal, the device comprising: The acquisition unit is used to acquire real-scene images of the vehicle; The generation unit is used to generate one or more interactive hotspots in the real-world image of the vehicle, wherein each interactive hotspot corresponds to a controllable part of the vehicle in the real-world image of the vehicle. The display unit is used to respond to the first operation on the interactive hotspot and display the vehicle control command panel, wherein the vehicle control command panel is the vehicle control command panel of the controllable part of the vehicle corresponding to the interactive hotspot. The control unit is used to control the vehicle in response to a second operation on the vehicle control command panel.

[0020] Thirdly, the present invention provides a mobile terminal, which includes: a memory, a transceiver, and a processor; Memory is used to store computer programs; transceiver is used to send and receive data under the control of the processor; processor is used to read the computer program from memory and execute it. Acquire real-view images of the vehicle; One or more interactive hotspots are generated in the real-world vehicle image, where each interactive hotspot corresponds to a controllable part of the vehicle in the real-world vehicle image; In response to the first operation on the interactive hotspot, the vehicle control command panel is displayed, which is the vehicle control command panel for the controllable part of the vehicle corresponding to the interactive hotspot. The vehicle is controlled in response to a second operation on the vehicle control command panel.

[0021] Fourthly, the present invention proposes a vehicle, which includes a mobile terminal as described in the third aspect.

[0022] Fifthly, the present invention provides a processor-readable storage medium storing a program for causing the processor to execute the vehicle control method of the first aspect.

[0023] The beneficial effects of this invention are: The mobile terminal acquires a real-world image of the vehicle; generates one or more interactive hotspots in the real-world image, each corresponding to a controllable part of the vehicle in the image; in response to a first operation on the interactive hotspot, displays a vehicle control command panel, which is a control command panel for the controllable part of the vehicle corresponding to the interactive hotspot; and in response to a second operation on the vehicle control command panel, controls the vehicle. This invention transforms the vehicle control entry point from an abstract menu to a real vehicle part captured by the user, achieving a WYSIWYG vehicle control experience, reducing the cost of finding vehicle control options, minimizing errors, and improving the accuracy and safety of vehicle control. Attached Figure Description

[0024] Figure 1 A schematic diagram of a vehicle control method provided by the present invention; Figure 2 A schematic diagram of a vehicle control device provided by the present invention; Figure 3 This is a schematic diagram of a mobile terminal provided by the present invention. Detailed Implementation

[0025] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0026] It should be noted that in this article, relational terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0027] This invention relates to intelligent vehicles, mobile terminal interaction, image recognition, and vehicle-to-everything (V2X) remote control technology, and particularly to a vehicle control method based on the recognition of vehicle parts in real-world images. This invention allows users to obtain corresponding vehicle control entry points simply by photographing the vehicle parts they see. It reduces the cost of finding vehicle control entry points without requiring users to understand complex function menus and vehicle part names, and accurately selects the target part when multiple parts are in the same frame, reducing the probability of misoperation.

[0028] Figure 1 This is a schematic flowchart of a vehicle control method provided by the present invention. This method is applied to a mobile terminal, which can be a mobile phone, tablet computer, smart wearable device, or other portable electronic device used by the vehicle owner. Figure 1 As shown, the vehicle control method includes steps 101 to 104: In step 101, a real-view image of the vehicle is acquired.

[0029] In this embodiment, a user can use a mobile terminal to take pictures of the vehicle from the outside or inside, and the mobile terminal then acquires the captured real-view images of the vehicle. For example, the mobile terminal provides a real-view vehicle control entry point. After entering the real-view vehicle control entry point through the mobile terminal's application, the user can take pictures of the exterior or interior of the vehicle, select previously captured images from the album, or, while scanning the vehicle, the mobile terminal automatically triggers the camera to capture an image based on a preset duration (e.g., 2 seconds) of camera stillness. Therefore, the mobile terminal can access the camera to acquire real-view images of the vehicle, or it can read locally selected images from the user's location.

[0030] In step 102, one or more interactive hotspots are generated in the real-world vehicle image, wherein each interactive hotspot corresponds to a controllable part of the vehicle in the real-world vehicle image.

[0031] The real-world image of a vehicle may include only one part or multiple parts at the same time.

[0032] In this embodiment, the mobile terminal can generate interactive hotspots at different controllable parts of the vehicle in the real-world vehicle image. The location of the interactive hotspots corresponds to the location of the controllable parts of the vehicle, allowing users to intuitively see the interactive hotspots corresponding to different controllable parts of the vehicle in the real-world vehicle image. Users can determine which parts of the vehicle are controllable without understanding complex function menus and vehicle part names. This reduces the user's cognitive cost and operating steps.

[0033] In some embodiments, the mobile terminal may generate one or more interactive hotspots based on the coordinate system of the vehicle real-scene image, and map one or more interactive hotspots to the screen display coordinate system of the mobile terminal.

[0034] In step 103, in response to the first operation on the interactive hotspot, a vehicle control command panel is displayed, wherein the vehicle control command panel is the vehicle control command panel of the controllable part of the vehicle corresponding to the interactive hotspot.

[0035] For example, interactive hotspots are clickable hotspots. Users only need to click the clickable hotspot corresponding to the controllable part of the vehicle they want to control, and the mobile terminal will respond to the click operation of the clickable hotspot and pop up the vehicle control command panel for the corresponding controllable part of the vehicle. The vehicle control command panel displays the vehicle control commands that the user can select, eliminating the need for the user to search for vehicle control commands from a complex function menu.

[0036] For example, when a user clicks on any clickable hotspot, the mobile terminal will pop up a vehicle control command panel for the controllable part of the vehicle corresponding to that hotspot, and display the name, real-time status, executable commands, risk warnings and execute button of the controllable part of the vehicle in the vehicle control command panel.

[0037] In step 104, the vehicle is controlled in response to a second operation on the vehicle control command panel.

[0038] For example, in response to a user's selection of a control command on the vehicle control command panel, the mobile terminal determines the selected control command; it then sends the control command to the server (e.g., a vehicle-to-everything (V2X) server), causing the server to distribute the control command to the vehicle for execution; subsequently, it receives the execution result of the control command sent by the server, which is then fed back from the vehicle to the server. The mobile terminal can display the execution result or execution status in an interactive hotspot and / or on the vehicle control command panel.

[0039] In some embodiments, if the user zooms in, zooms out, or rotates the real-view image of the vehicle, the hotspot coordinates are updated synchronously with the transformation matrix of the real-view image of the vehicle to ensure that the hotspot always fits the corresponding controllable part of the vehicle.

[0040] As can be seen, in this embodiment, the mobile terminal acquires a real-world image of the vehicle; generates one or more interactive hotspots in the real-world image, where each interactive hotspot corresponds to a controllable part of the vehicle in the real-world image; in response to a first operation on the interactive hotspot, displays a vehicle control command panel, where the vehicle control command panel is the control command panel for the controllable part of the vehicle corresponding to the interactive hotspot; and in response to a second operation on the vehicle control command panel, controls the vehicle. In this embodiment, the vehicle control entry point is transformed from an abstract menu into a real vehicle part photographed by the user, allowing the user to obtain the corresponding vehicle control entry point by photographing the vehicle part they see, achieving a WYSIWYG vehicle control experience. This reduces the cost of finding vehicle control options without requiring the user to understand complex function menus and vehicle part names, and by operating on the interactive hotspots, it reduces misoperations and improves the accuracy and safety of vehicle control.

[0041] In some embodiments, after acquiring the real-scene image of the vehicle in step 101, the method further includes: performing quality detection on the real-scene image of the vehicle, wherein the quality detection includes multiple detection items; if any detection item does not meet the detection threshold corresponding to the detection item, the user is prompted to reacquire the real-scene image of the vehicle (e.g., prompting the user to retake the photo or adjust the shooting angle). The multiple detection items include: sharpness detection, exposure detection, occlusion detection, target proportion detection, and vehicle-related region detection; wherein, target proportion detection is used to detect the proportion of the vehicle area in the real-scene image of the vehicle; and vehicle-related region detection is used to determine whether a vehicle exists in the real-scene image of the vehicle.

[0042] It is evident that by performing quality inspection on real-world vehicle images, we can ensure that the images meet the recognition requirements, avoid recognition errors caused by image quality issues, and improve the reliability of recognizing controllable parts of the vehicle.

[0043] In some embodiments, before step 102 is performed, the method further includes: identifying controllable parts of the vehicle in the real-world image of the vehicle, and obtaining one or more identification results, wherein each identification result includes a part category of a controllable part of the vehicle and location information of the controllable part of the vehicle.

[0044] For example, a real-world image of a vehicle is input into a pre-trained vehicle part recognition model for image processing, resulting in one or more recognition results output by the vehicle part recognition model. The vehicle part recognition model can employ any of the following models: object detection model, semantic segmentation model, instance segmentation model, or multi-task recognition model.

[0045] Among them, the object detection model can quickly detect vehicle parts in an image and output detection boxes; the semantic segmentation model and the instance segmentation model can output pixel-level segmentation masks to more accurately represent the contours of the parts; the multi-task recognition model can simultaneously output multiple information such as detection boxes, segmentation masks, and key points.

[0046] As can be seen, the vehicle part recognition model can automatically identify controllable parts of a vehicle in an image, eliminating the need for users to manually select them on the virtual model. By automatically identifying controllable parts of the vehicle through image recognition technology and generating interactive hotspots in the real-world vehicle image, it reduces the cognitive cost and operational steps for users, eliminating the need for them to understand abstract virtual models.

[0047] In some embodiments, the vehicle part recognition model can be deployed on the mobile terminal locally, on a cloud server, or in an edge-cloud collaborative architecture that combines both.

[0048] In some embodiments, the position of the interactive hotspot in the vehicle real-world image is determined by the position information of the controllable part of the vehicle corresponding to the interactive hotspot. The position information of the controllable part of the vehicle includes a detection bounding box or a segmentation mask. A detection bounding box is suitable for vehicle parts with regular shapes. A segmentation mask is suitable for irregularly shaped or elongated parts, and can more accurately represent the spatial position and contour of the vehicle part.

[0049] In some embodiments, the interactive hotspot includes an interactive origin and an interactive radius. The interactive origin can be any of the following: The center of the detection frame, the center of the segmentation mask, and the preset interaction priority point are determined in advance based on the part category of the controllable parts of the vehicle.

[0050] For example, interaction priority points are fixed coordinates pre-marked in the vehicle configuration table and bound to the vehicle model. They are directly invoked after the corresponding part is identified. Example: For vehicle parts with clearly defined operating areas, such as doors, trunks, and charging ports, the center or edge opening area of ​​the handle or cover can be used as the interaction priority point.

[0051] For example, for regularly shaped parts (such as car windows), the interactive origin is located at the center of the detection frame; for long and narrow parts (such as car door handles), the interactive origin is located at the center of the skeleton of the segmentation mask; for parts with a clear operating area (such as the trunk), the interactive origin is located at a preset interaction priority point (such as the position of the trunk opening button).

[0052] In some embodiments, the interactive radius is determined based on the area of ​​the controllable part of the vehicle, the screen size, and the distance between adjacent parts.

[0053] For example, the area of ​​the controllable part of the vehicle, the screen size, and the distance between adjacent parts can be weighted and summed based on the part area coefficient, screen scaling coefficient, and distance avoidance coefficient, and then superimposed on the preset base radius to obtain the interactive radius. This allows for dynamic adjustment of the interactive radius corresponding to different controllable parts of the vehicle, ensuring that the size of the interactive hotspots is appropriate and they do not overlap.

[0054] In some embodiments, if the distance between two adjacent interactive hotspots is less than a preset distance threshold, the positions of the two adjacent interactive hotspots are adjusted, or the local area where the two adjacent interactive hotspots are located is enlarged.

[0055] For example, when the distance between two adjacent interactive hotspots is less than twice the interactive radius, the two adjacent interactive hotspots are rearranged to avoid each other, or a local area of ​​the vehicle's real-world image is magnified. The hotspot avoidance mechanism prevents accidental touches caused by hotspot overlap when multiple parts are in the same frame and close together, thus improving interaction accuracy.

[0056] In some embodiments, the identification result further includes: controllable attributes, which are used to identify whether controllable parts of the vehicle support remote control. In step 102, one or more interactive hotspots are generated in the vehicle real-world image, including: For any recognition result, if the controllable attribute in the recognition result indicates that the controllable part of the vehicle supports remote control, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

[0057] As can be seen, by filtering out uncontrollable parts of the vehicle in the real-world image by controlling attributes, interactive hotspots are generated only for controllable parts, thus preventing the system from failing to respond when a user clicks on an uncontrollable part and improving the interactive experience.

[0058] In some embodiments, the identification result further includes: confidence level, which is used to identify the degree of credibility of controllable parts of the vehicle. In step 102, one or more interactive hotspots are generated in the real-world image of the vehicle, including the following (1) and (2): (1) For any recognition result, if the confidence level in the recognition result is greater than or equal to the first confidence level threshold, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

[0059] (2) If the confidence level in the identification result is less than the first confidence threshold and greater than or equal to the second confidence threshold, then a hot spot to be confirmed corresponding to the controllable part of the vehicle is generated, wherein the first confidence threshold is greater than the second confidence threshold.

[0060] For example, when the recognition confidence of a controllable part of the vehicle is lower than the second confidence threshold, no interactive hotspot is generated, and the user is prompted to reacquire a real-world image of the vehicle. When the recognition confidence is less than the first confidence threshold but greater than or equal to the second confidence threshold, a hotspot to be confirmed is generated. In response to the user's click on the hotspot, a list of candidate parts is displayed for the user to select, and the vehicle control command is displayed after the user selects the specific part. When the recognition confidence is greater than or equal to the first confidence threshold, an interactive hotspot is generated directly. It is evident that the low-confidence processing mechanism improves the reliability of vehicle control and avoids miscontrol due to recognition uncertainty through confidence-based tiered processing.

[0061] For example, when only one controllable vehicle part meets the confidence threshold and controllability attribute in the recognition results, the mobile terminal directly pops up the vehicle control command panel corresponding to that controllable part. When multiple controllable vehicle parts meet the confidence threshold and controllability attribute in the recognition results, the mobile terminal displays multiple interactive hotspots on the vehicle's real-world image. It is evident that the interaction method is adaptively selected based on the number of recognition results. For a single part, the operation process is simplified; for multiple parts, the target part is accurately selected through hotspots, reducing misoperations.

[0062] For example, mobile terminals can combine vehicle 3D models or 2D part topology tables to verify recognition results. For instance, when both the left front door and right rear door are identified in an image, but their spatial relationship is inconsistent with the current shooting perspective, the recognition result can be corrected or the confidence level reduced based on the vehicle's part topology, shooting direction estimation, and relationships with adjacent parts. This method can reduce the risk of miscontrol caused by misidentification of similar parts.

[0063] In some embodiments, the control commands in the vehicle control command panel are determined by the part type of the controllable vehicle part corresponding to the interactive hotspot. For example: when the part type is a door, the control commands include at least one of unlocking, locking, checking the status of the child lock, and a door not closed reminder; when the part type is a window, the control commands include at least one of raising the window, lowering the window, and ventilation; when the part type is the trunk, the control commands include at least one of opening, closing, and checking the status; when the part type is the charging port cover, the control commands include at least one of opening the charging port, closing the charging port, and checking the charging status; when the part type is an air conditioning vent or the in-vehicle air conditioning area, the control commands include at least one of turning on the air conditioning, adjusting the temperature, defogging, or pre-cooling / pre-heating the cabin. These commands directly correspond to the functions of the parts, allowing users to quickly understand and execute them.

[0064] In some embodiments, the recognition result further includes: a scene identifier, which is used to identify whether the real-world image of the vehicle is taken from outside the vehicle or from inside the vehicle. The method further includes: dynamically adjusting the control commands in the vehicle control command panel or adjusting the execution risk level of the control commands based on the scene identifier.

[0065] For example, in exterior vehicle scenarios (where scene identifiers indicate that the vehicle's real-world image was taken from outside the vehicle), commands for door locks, windows, trunk, charging port, vehicle location, lights, and air conditioning pre-cooling / pre-heating are prioritized. In interior vehicle scenarios (where scene identifiers indicate that the vehicle's real-world image was taken from inside the vehicle), commands for air conditioning, seats, windows, sunroof, ambient lighting, center console screen, and steering wheel heating are prioritized. It is evident that dynamically adjusting controllable commands through scene identifiers can improve the relevance and safety of these commands.

[0066] For example, when the same controllable part of a vehicle corresponds to different command risk levels in external and internal vehicle scenarios, the execution risk level of the control command can be dynamically adjusted based on the scenario identifier, such as requiring secondary confirmation in the internal vehicle scenario.

[0067] In some embodiments, step 104, in response to a first operation on the interactive hotspot, displays a vehicle control command panel, including: in response to the first operation on the interactive hotspot, obtaining vehicle status information from the server; based on the vehicle status information, determining executable commands for the controllable parts of the vehicle corresponding to the interactive hotspot; and displaying the vehicle control command panel, wherein the vehicle control command panel displays the executable commands.

[0068] For example, the vehicle control commands in the control command panel are generated based on a location-command-status-safety policy mapping table. Executable commands are then filtered from the mapping table based on vehicle configuration and real-time vehicle status. The mapping table includes at least one of the following fields: location identifier, command identifier, command name, applicable vehicle model, applicable scenario, execution prerequisites, risk level, whether secondary confirmation is required, whether remote execution is supported, and execution result display. The mapping table is queried based on the identified location category, and commands are dynamically filtered by combining vehicle configuration (e.g., whether the vehicle model supports remote trunk opening) and real-time status (e.g., vehicle gear and speed) to avoid displaying unexecutable control items, thus improving command adaptability and security.

[0069] For example, when the left front door is detected, if the vehicle is parked and the user has control authority, commands such as unlocking, locking, and checking the door status can be displayed; if the vehicle is in motion, high-risk commands such as remote unlocking can be hidden or only the status can be displayed.

[0070] In some embodiments, the categories of controllable parts of the vehicle include at least one of the following: doors (e.g., left front door, right front door, left rear door, right rear door), windows, trunk, hood, charging port cover, fuel tank cap, exterior rearview mirrors, rear seats, lights, sunroof, air conditioning vents, seats, steering wheel, central control screen, instrument panel, and gear shift mechanism. These parts cover the main controllable parts of the vehicle's exterior and interior, meeting the user's vehicle control needs in different scenarios.

[0071] In some embodiments, the recognition results can be further calibrated by combining the Vehicle Identification Number (VIN), vehicle configuration table, vehicle color, vehicle posture, and shooting angle. Vehicle calibration involves querying the vehicle configuration table using the vehicle VIN to obtain the layout, color, and appearance features of the vehicle model, and then using the vehicle posture and shooting angle to correct or reduce the confidence level of the recognition results, thereby reducing the risk of miscontrol due to misidentification of similar parts.

[0072] In some embodiments, before the vehicle control command is sent from the vehicle network server to the vehicle for execution, or before the user confirms the execution of the vehicle control command, the mobile terminal or the vehicle network server can perform security verification on the user's identity, vehicle binding relationship, vehicle online status, vehicle gear position, vehicle speed, door and window status, in-vehicle living being detection status, Bluetooth or network connection status, geofencing, and command risk level. Once the security verification is successful, the mobile terminal sends the vehicle control command to the vehicle network server, or the vehicle network server sends the vehicle control command to the vehicle for execution. Therefore, this multi-dimensional security verification mechanism ensures the security of vehicle control operations and reduces misoperation and security risks.

[0073] In some embodiments, safety verification includes: prohibiting remote door opening commands when the vehicle speed exceeds a preset speed threshold or the vehicle is not in park; and restricting or requiring secondary confirmation of window closing, vehicle locking, and air conditioning shut-off commands when a child or pet is detected inside the vehicle. The safety verification dynamically adjusts executable commands based on the vehicle's real-time status, for example, preventing remote door opening while the vehicle is in motion and preventing accidental window closing when children or pets are inside the vehicle.

[0074] For example, the security verification process includes: the mobile terminal receiving confirmation from the user for a vehicle control command; the mobile terminal verifying the user account, vehicle binding relationship, digital key, or remote vehicle control permission; the mobile terminal obtaining the real-time vehicle status from the vehicle network server or vehicle control module, including vehicle online status, power status, gear position, vehicle speed, door and window status, lock status, charging status, and cabin life detection status; the mobile terminal determining whether the command is allowed to be executed according to preset security policies, such as prohibiting remote door opening while the vehicle is in motion, and restricting or requiring secondary confirmation for commands such as closing windows, locking the car, or turning off the air conditioning when a child or pet is detected inside the vehicle; the mobile terminal sending the vehicle control command after successful verification; and the mobile terminal receiving the execution result from the vehicle and displaying it on the mobile terminal.

[0075] Based on the description of the above embodiments, the present invention has at least the following differences from the prior art: (1) The vehicle controllable parts are identified based on real-world photos of the vehicle interior / exterior taken by the vehicle owner’s mobile terminal, rather than relying on fixed cameras inside the vehicle or pre-set vehicle model pages.

[0076] (2) The recognition results not only output the category of the part, but also the location range, confidence level and controllable attributes of the part in the original photo, and based on this, interactive hotspots are superimposed on the photo.

[0077] (3) When there is only one controllable part in the photo, the vehicle control command panel for that part will automatically pop up; when there are multiple controllable parts, multiple hotspots will be generated, and the corresponding command panel will pop up after the user clicks on any hotspot.

[0078] (4) Set up a mapping table of “location-instruction-status-security policy” to filter executable instructions based on vehicle VIN, vehicle model configuration, current status, user permissions, vehicle interior and exterior scenarios and security level to avoid displaying unavailable or high-risk instructions.

[0079] (5) Supports image quality detection, viewpoint / interior / exterior scene recognition, part disambiguation, low confidence secondary confirmation, pre-execution status verification and execution result feedback, forming a closed loop from shooting recognition to vehicle control feedback.

[0080] Based on the description of the above embodiments, the present invention has at least the following technical effects: First, the vehicle control entry point is changed from an abstract menu to a real vehicle part photographed by the user, reducing the search cost.

[0081] Second, when multiple parts are in the same frame, target selection is performed using hotspots in the original image, reducing accidental operations.

[0082] Third, based on vehicle configuration and real-time status, commands are dynamically filtered to avoid displaying unexecutable vehicle control items.

[0083] Fourth, improve the reliability of the vehicle identification and control link through image quality detection, confidence control, and security verification.

[0084] Fifth, the same solution is compatible with both exterior and interior vehicle scenarios, and is suitable for scenarios where vehicle parts are complex, users are unfamiliar with function names, or need quick vehicle control. For example, it is suitable for commonly used functions that are often hidden, such as rearview mirror adjustment and rear seat adjustment.

[0085] Sixth, this invention is also applicable to scenarios such as real-time video recognition, augmented reality vehicle control, voice and image-based interaction, vehicle fault description, function instruction, and after-sales service. For example, when a vehicle is involved in a scratch or malfunction, the corresponding function options can be quickly displayed by identifying the problematic area, making it convenient for users to handle emergency situations.

[0086] Examples of emergency scenarios are illustrated below: When a minor collision occurs while driving or parking, or when a sudden vehicle malfunction alarm sounds, this invention can use the mobile terminal's camera to identify the damaged or faulty parts in real time (such as bumper scratches, low tire pressure, smoke coming from the engine compartment, etc.). Upon identification, the mobile terminal's screen immediately displays targeted function options, such as the following targeted function options A to D: A. Accident handling: Provides guidance on taking photos for evidence, one-click call to insurance companies or traffic police, and records the scene location and timestamp; B. Troubleshooting: Displays common causes of malfunctions in this area, temporary solutions (such as how to safely pull over, use the spare tire or tire sealant), and automatically pushes navigation to the nearest authorized service station; C. After-sales appointment: Initiate remote diagnosis or schedule in-store repair with one click, and upload on-site images simultaneously for technicians to make a preliminary judgment; D. Teaching assistance: Play a short video explaining the structure of this part, or use AR overlay to show simple operations such as checking the oil and changing the light bulb.

[0087] The entire process does not require users to manually search the menu. A single visual recognition can connect the complete closed loop from emergency handling to follow-up services, greatly improving the efficiency and sense of security for users in emergency situations.

[0088] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art will understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in the specification are all optional embodiments.

[0089] Figure 2 This is a schematic diagram of a vehicle control device provided by the present invention. This device is applied to a mobile terminal, such as... Figure 2 As shown, the device includes: Acquisition unit 21 is used to acquire real-scene images of the vehicle; The generation unit 22 is used to generate one or more interactive hotspots in the real-world vehicle image, wherein each interactive hotspot corresponds to a controllable part of the vehicle in the real-world vehicle image. Display unit 23 is used to respond to the first operation on the interactive hotspot and display the vehicle control command panel, wherein the vehicle control command panel is the vehicle control command panel of the controllable part of the vehicle corresponding to the interactive hotspot. Control unit 24 is used to control the vehicle in response to a second operation on the vehicle control command panel.

[0090] In some embodiments, the device further includes a quality detection unit for: The quality inspection of real-world vehicle images includes multiple inspection items. If any detection item fails to meet the corresponding detection threshold, a prompt will be made to reacquire the real-world image of the vehicle.

[0091] In some embodiments, the plurality of detection items include: The system includes sharpness detection, exposure detection, occlusion detection, target proportion detection, and vehicle-related region detection. Among these, target proportion detection is used to detect the area of ​​a vehicle in a real-world vehicle image, while vehicle-related region detection is used to determine whether a vehicle exists in the real-world vehicle image.

[0092] In some embodiments, the device further includes an identification unit for: Identify controllable parts of a vehicle in a real-world image and obtain one or more identification results. Each identification result includes the part category of a controllable part and the location information of the controllable part.

[0093] In some embodiments, the identification unit is used for: The real-world image of the vehicle is input into a pre-trained vehicle part recognition model for image processing, resulting in one or more recognition results output by the vehicle part recognition model.

[0094] In some embodiments, the position of the interactive hotspot in the real-world image of the vehicle is determined by the position information of the controllable part of the vehicle corresponding to the interactive hotspot.

[0095] In some embodiments, the location information of the controllable part of the vehicle includes: a detection frame or a segmentation mask; the interactive hotspot includes: an interactive origin and an interactive radius; The interactive origin can be any of the following: The center of the detection frame; The center of the segmented mask; Preset interaction priority points are determined in advance based on the category of controllable parts of the vehicle.

[0096] In some embodiments, the interactive radius is determined based on the area of ​​the controllable part of the vehicle, the screen size, and the distance between adjacent parts.

[0097] In some embodiments, the generating unit 22 is further configured to: adjust the position of the two adjacent interactive hotspots if the distance between two adjacent interactive hotspots is less than a preset distance threshold, or enlarge the local area where the two adjacent interactive hotspots are located.

[0098] In some embodiments, the identification result further includes: controllable attributes, which are used to identify whether the controllable parts of the vehicle support remote control; Generation unit 22 is used for: For any recognition result, if the controllable attribute in the recognition result indicates that the controllable part of the vehicle supports remote control, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

[0099] In some embodiments, the identification result further includes: confidence level, which is used to identify the degree of trustworthiness of controllable parts of the vehicle; Generation unit 22 is used for: For any recognition result, if the confidence level in the recognition result is greater than or equal to the first confidence level threshold, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

[0100] In some embodiments, the generating unit 22 is further configured to: If the confidence level in the identification result is less than the first confidence threshold but greater than or equal to the second confidence threshold, then a hot spot to be confirmed corresponding to the controllable part of the vehicle is generated, wherein the first confidence threshold is greater than the second confidence threshold.

[0101] In some embodiments, the control commands in the vehicle control command panel are determined by the part type of the controllable part of the vehicle corresponding to the interactive hotspot.

[0102] In some embodiments, the recognition result further includes: scene identifier, which is used to identify whether the real-world image of the vehicle is taken from outside the vehicle or inside the vehicle; The device also includes an adjustment unit for adjusting the control commands in the vehicle control command panel or adjusting the execution risk level of the control commands based on scene identifiers.

[0103] In some embodiments, the display unit 23 is used for: In response to the first action on the interactive hotspot, vehicle status information is obtained from the server; Based on vehicle status information, determine the executable commands for the controllable parts of the vehicle corresponding to the interactive hotspots; The vehicle control command panel is displayed, showing the executable commands.

[0104] In some embodiments, the control unit 24 is used to: In response to the second operation on the vehicle control command panel, confirm the vehicle control command; Send vehicle control commands to the server; Receive the execution result of the vehicle control command sent by the server.

[0105] Figure 2 For details of the various embodiments of the vehicle control device shown, please refer to... Figure 1 The various embodiments of the vehicle control method shown are not described again to avoid repetition.

[0106] Figure 3This is a schematic diagram of a mobile terminal provided by the present invention. (See diagram below.) Figure 3 As shown, the mobile terminal provided by the present invention includes a memory 31, a transceiver 32, and a processor 33. Memory 31 is used to store computer programs; transceiver 32 is used to send and receive data under the control of processor 33; processor 33 is used to read the computer program in memory 31 and execute it. Acquire real-view images of the vehicle; One or more interactive hotspots are generated in the real-world vehicle image, where each interactive hotspot corresponds to a controllable part of the vehicle in the real-world vehicle image; In response to the first operation on the interactive hotspot, the vehicle control command panel is displayed, which is the vehicle control command panel for the controllable part of the vehicle corresponding to the interactive hotspot. The vehicle is controlled in response to a second operation on the vehicle control command panel.

[0107] In some embodiments, after acquiring the real-view image of the vehicle, the processor 33 is further configured to: The quality inspection of real-world vehicle images includes multiple inspection items. If any detection item fails to meet the corresponding detection threshold, a prompt will be made to reacquire the real-world image of the vehicle.

[0108] In some embodiments, the plurality of detection items include: The system includes sharpness detection, exposure detection, occlusion detection, target proportion detection, and vehicle-related region detection. Among these, target proportion detection is used to detect the area of ​​a vehicle in a real-world vehicle image, while vehicle-related region detection is used to determine whether a vehicle exists in the real-world vehicle image.

[0109] In some embodiments, the processor 33 is further configured to: Identify controllable parts of a vehicle in a real-world image and obtain one or more identification results. Each identification result includes the part category of a controllable part and the location information of the controllable part.

[0110] In some embodiments, the controllable parts of a vehicle in a real-world vehicle image are identified to obtain one or more identification results, including: The real-world image of the vehicle is input into a pre-trained vehicle part recognition model for image processing, resulting in one or more recognition results output by the vehicle part recognition model.

[0111] In some embodiments, the position of the interactive hotspot in the real-world image of the vehicle is determined by the position information of the controllable part of the vehicle corresponding to the interactive hotspot.

[0112] In some embodiments, the location information of the controllable part of the vehicle includes: a detection frame or a segmentation mask; the interactive hotspot includes: an interactive origin and an interactive radius; The interactive origin can be any of the following: The center of the detection frame; The center of the segmented mask; Preset interaction priority points are determined in advance based on the category of controllable parts of the vehicle.

[0113] In some embodiments, the interactive radius is determined based on the area of ​​the controllable part of the vehicle, the screen size, and the distance between adjacent parts.

[0114] In some embodiments, if the distance between two adjacent interactive hotspots is less than a preset distance threshold, the positions of the two adjacent interactive hotspots are adjusted, or the local area where the two adjacent interactive hotspots are located is enlarged.

[0115] In some embodiments, the identification result further includes: controllable attributes, which are used to identify whether the controllable parts of the vehicle support remote control; Generate one or more interactive hotspots in the real-world image of the vehicle, including: For any recognition result, if the controllable attribute in the recognition result indicates that the controllable part of the vehicle supports remote control, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

[0116] In some embodiments, the identification result further includes: confidence level, which is used to identify the degree of trustworthiness of controllable parts of the vehicle; Generate one or more interactive hotspots in the real-world image of the vehicle, including: For any recognition result, if the confidence level in the recognition result is greater than or equal to the first confidence level threshold, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

[0117] In some embodiments, the processor 33 is further configured to: If the confidence level in the identification result is less than the first confidence threshold but greater than or equal to the second confidence threshold, then a hot spot to be confirmed corresponding to the controllable part of the vehicle is generated, wherein the first confidence threshold is greater than the second confidence threshold.

[0118] In some embodiments, the control commands in the vehicle control command panel are determined by the part type of the controllable part of the vehicle corresponding to the interactive hotspot.

[0119] In some embodiments, the recognition result further includes: scene identifier, which is used to identify whether the real-world image of the vehicle is taken from outside the vehicle or inside the vehicle; The processor 33 is also used to: adjust the control commands in the vehicle control command panel or adjust the execution risk level of the control commands based on the scene identifier.

[0120] In some embodiments, in response to a first operation on an interactive hotspot, a vehicle control command panel is displayed, including: In response to the first action on an interactive hotspot, vehicle status information is obtained from the server. Based on vehicle status information, determine the executable commands for the controllable parts of the vehicle corresponding to the interactive hotspots; The vehicle control command panel is displayed, showing the executable commands.

[0121] In some embodiments, controlling the vehicle in response to a second operation on the vehicle control command panel includes: In response to the second operation on the vehicle control command panel, confirm the vehicle control command; Send vehicle control commands to the server; Receive the execution result of the vehicle control command sent by the server.

[0122] Figure 3 For details of the various embodiments of the mobile terminal shown, please refer to Figure 1 The various embodiments of the vehicle control method shown are not described again to avoid repetition.

[0123] Based on the same technical concept, the present invention also proposes a vehicle, including as follows: Figure 3 The provided mobile terminal.

[0124] Based on the same technical concept, the present invention also provides a processor-readable storage medium storing a program for causing a processor to execute the steps of various embodiments of the vehicle control method. The processor-readable storage medium can be any available medium or data storage device accessible to the processor, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical memory (e.g., CD, DVD, BD, HVD, etc.), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD), etc.).

[0125] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A vehicle control method, characterized in that, Applied to a mobile terminal, the method includes: Acquire real-view images of the vehicle; One or more interactive hotspots are generated in the real-world vehicle image, wherein each interactive hotspot corresponds to a controllable part of the vehicle in the real-world vehicle image; In response to a first operation on the interactive hotspot, a vehicle control command panel is displayed, wherein the vehicle control command panel is the vehicle control command panel of the controllable part of the vehicle corresponding to the interactive hotspot. In response to a second operation on the vehicle control command panel, the vehicle is controlled.

2. The method according to claim 1, characterized in that, After acquiring the real-view image of the vehicle, the method further includes: The real-scene image of the vehicle is subjected to quality inspection, wherein the quality inspection includes multiple inspection items; If any of the detection items does not meet the detection threshold corresponding to the detection item, a prompt will be made to reacquire the real-world image of the vehicle.

3. The method according to claim 2, characterized in that, The multiple detection items include: The system includes sharpness detection, exposure detection, occlusion detection, target proportion detection, and vehicle-related region detection; wherein, the target proportion detection is used to detect the area proportion of the vehicle in the real-world vehicle image; and the vehicle-related region detection is used to determine whether a vehicle exists in the real-world vehicle image.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Identify controllable parts of the vehicle in the real-world image of the vehicle to obtain one or more identification results, wherein each identification result includes a part category of a controllable part of the vehicle and the location information of the controllable part of the vehicle.

5. The method according to claim 4, characterized in that, The process of identifying controllable parts of the vehicle in the real-world image of the vehicle, and obtaining one or more identification results, includes: The real-world image of the vehicle is input into a pre-trained vehicle part recognition model for image processing to obtain one or more recognition results output by the vehicle part recognition model.

6. The method according to claim 4, characterized in that, The position of the interactive hotspot in the real-world image of the vehicle is determined by the position information of the controllable part of the vehicle corresponding to the interactive hotspot.

7. The method according to claim 6, characterized in that, The location information of the controllable parts of the vehicle includes: a detection frame or a segmentation mask; the interactive hotspot includes: an interactive origin and an interactive radius. The interactive origin can be any of the following: The center of the detection frame; The center of the segmentation mask; A preset interaction priority point, which is predetermined based on the part category of the controllable part of the vehicle.

8. The method according to claim 7, characterized in that, The interactive radius is determined based on the area of ​​the controllable part of the vehicle, the screen size, and the distance between adjacent parts.

9. The method according to claim 8, characterized in that, If the distance between two adjacent interactive hotspots is less than a preset distance threshold, the positions of the two adjacent interactive hotspots are adjusted, or the local area where the two adjacent interactive hotspots are located is enlarged.

10. The method according to claim 4, characterized in that, The identification result also includes: controllable attributes, which are used to identify whether the controllable parts of the vehicle support remote control; The step of generating one or more interactive hotspots in the real-world image of the vehicle includes: For any of the recognition results, if the controllable attribute in the recognition result indicates that the controllable part of the vehicle supports remote control, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

11. The method according to claim 4, characterized in that, The identification result also includes: confidence level, which is used to identify the degree of trustworthiness of the controllable parts of the vehicle; The step of generating one or more interactive hotspots in the real-world image of the vehicle includes: For any of the recognition results, if the confidence level in the recognition result is greater than or equal to the first confidence level threshold, then an interactive hotspot corresponding to the controllable part of the vehicle is generated.

12. The method according to claim 11, characterized in that, The method further includes: If the confidence level in the identification result is less than the first confidence threshold but greater than or equal to the second confidence threshold, then a hot spot to be confirmed corresponding to the controllable part of the vehicle is generated, wherein the first confidence threshold is greater than the second confidence threshold.

13. The method according to claim 4, characterized in that, The control commands in the vehicle control command panel are determined by the part type of the controllable part of the vehicle corresponding to the interactive hotspot.

14. The method according to claim 4, characterized in that, The recognition result also includes: scene identification, which is used to identify whether the real-scene image of the vehicle is taken from outside the vehicle or inside the vehicle; The method further includes: adjusting the control commands in the vehicle control command panel or adjusting the execution risk level of the control commands based on the scenario identifier.

15. The method according to claim 1, characterized in that, The response to the first operation on the interactive hotspot, displaying the vehicle control command panel, includes: In response to a first operation targeting the interactive hotspot, vehicle status information is obtained from the server. Based on the vehicle status information, determine the executable commands for the controllable parts of the vehicle corresponding to the interactive hotspot; A vehicle control command panel is displayed, in which the executable commands are shown.

16. The method according to claim 1, characterized in that, The control of the vehicle in response to a second operation on the vehicle control command panel includes: In response to a second operation on the vehicle control command panel, a vehicle control command is determined; Send the vehicle control command to the server; Receive the execution result of the vehicle control command sent by the server.

17. A vehicle control device, characterized in that, The device, applied to a mobile terminal, includes: The acquisition unit is used to acquire real-scene images of the vehicle; The generation unit is used to generate one or more interactive hotspots in the real-world image of the vehicle, wherein each interactive hotspot corresponds to a controllable part of the vehicle in the real-world image of the vehicle. The display unit is used to display a vehicle control command panel in response to a first operation on the interactive hotspot, wherein the vehicle control command panel is the vehicle control command panel of the controllable part of the vehicle corresponding to the interactive hotspot. The control unit is used to control the vehicle in response to a second operation on the vehicle control command panel.

18. A mobile terminal, characterized in that, The mobile terminal includes: a memory, a transceiver, and a processor; The memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; the processor is used to read the computer programs in the memory and execute them. Acquire real-view images of the vehicle; One or more interactive hotspots are generated in the real-world vehicle image, wherein each interactive hotspot corresponds to a controllable part of the vehicle in the real-world vehicle image; In response to a first operation on the interactive hotspot, a vehicle control command panel is displayed, wherein the vehicle control command panel is the vehicle control command panel of the controllable part of the vehicle corresponding to the interactive hotspot. In response to a second operation on the vehicle control command panel, the vehicle is controlled.

19. A vehicle, characterized in that, The vehicle includes the mobile terminal as described in claim 18.

20. A processor-readable storage medium, characterized in that, The processor-readable storage medium stores a program for causing the processor to perform the vehicle control method as described in any one of claims 1 to 16.

Images