Central control display content customization method and system based on image processing

Abstract

The present application relates to the field of image processing technology, and discloses a method and system for customizing the display content of a central control based on image processing, the method comprising: obtaining the driving route of the vehicle, obtaining the current position of the vehicle; predicting multiple locations that the vehicle will pass through based on the driving route and current position of the vehicle; taking an image of the user in the vehicle; identifying the user's behavioral characteristics based on the user's image; calculating the user's interest in multiple locations based on the user's behavioral characteristics; selecting a target location from multiple locations; obtaining content related to the target location, and displaying it through the vehicle's central control system. According to the technical solution of the present invention, the user's interests and intentions can be inferred based on the user's facial expressions, line of sight direction, gesture type and other behavioral characteristics, so that the central control system can provide more personalized information and services to meet the user's specific needs.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a method and system for customizing central control display content based on image processing. Background Art

[0002] With the rapid development of smart vehicles and Internet of Vehicles technology, the vehicle central control system has become an indispensable part of modern cars. The central control system not only provides functions such as navigation, entertainment, and communication, but also provides customized services based on the personalized needs of users. However, the existing central control system has some limitations in content display, especially in customizing the display content based on the user's behavioral characteristics. Due to the lack of in-depth understanding of user behavior, the content provided by the central control system may not match the user's actual needs, resulting in a poor user experience. For example, a user may be interested in a certain place, but the system does not provide relevant information; or the system provides too much information, causing information overload. Excessive or irrelevant information display may distract the driver and affect driving safety.

[0003] Therefore, a new technical solution is needed that can intelligently perceive the intentions of users in the vehicle and display content that meets user needs through the central control system. Summary of the invention

[0004] In order to solve the above-mentioned technical problems, the present application is proposed to provide a central control display content customization method and system based on image processing, which can intelligently perceive the intention of the user in the vehicle, so as to display content that meets the user's needs through the central control system.

[0005] In a first aspect, the present application provides a method for customizing central control display content based on image processing, comprising: obtaining a vehicle's driving route and obtaining the vehicle's current position; predicting multiple locations that the vehicle will pass through based on the vehicle's driving route and current position; capturing an image of a user in the vehicle; identifying the user's behavioral characteristics based on the user's image; calculating the user's interest level in the multiple locations based on the user's behavioral characteristics; selecting a target location from the multiple locations based on the user's interest level in the multiple locations; obtaining content related to the target location and displaying it through the vehicle's central control system.

[0006] Optionally, the aforementioned central control display content customization method based on image processing identifies behavioral characteristics of the user according to the image of the user, including: when there are multiple users in the vehicle, identifying the identities of the multiple users from the images that have been taken containing the multiple users; analyzing the relationship between the multiple users based on the identities of the multiple users; selecting a user from the multiple users based on the relationship between the multiple users, and analyzing the behavioral characteristics of the selected user.

[0007] Optionally, the aforementioned method for customizing the central control display content based on image processing, identifying the user's behavioral characteristics according to the user's image, includes: identifying the user's head from the user's image; detecting the user's eye center position from the user's head position; and pupil center , and analyzing the deflection angles of the user's head in the horizontal and vertical directions ; Calculate the user's line of sight direction , as the behavioral characteristics of the user.

[0008] Optionally, in the aforementioned central control display content customization method based on image processing, the analysis of the deflection angle of the user's head in the horizontal direction and the vertical direction , including: detecting the user's facial key points from the user's head, wherein the coordinates of the i-th facial key point are ; Establish projection error function ,in, The function is the projection function from three-dimensional space to two-dimensional plane. is the i-th facial key point on the preset standard facial model, , They are rotation matrix and translation matrix respectively, and solve the projection error function The rotation matrix is ​​minimized , the translation matrix The value of the rotation matrix Decompose to obtain the deflection angle of the user's head in the horizontal and vertical directions .

[0009] Optionally, the aforementioned central control display content customization method based on image processing identifies the user's behavioral characteristics based on the user's image, including: identifying the user's hand area from the user's image; performing lighting compensation on the user's hand area in a preset manner; and identifying the user's gestures based on the user's hand area after lighting compensation as the user's behavioral characteristics.

[0010] Optionally, the aforementioned method for customizing the central control display content based on image processing performs illumination compensation on the user's hand area in a preset manner, including: detecting the illumination intensity of each pixel in the user's hand area; calculating the average illumination intensity of the user's hand area; performing illumination compensation on each pixel in the user's hand area, wherein the pixel value of the jth pixel after illumination compensation is ,in is the preset target light brightness, is the pixel value of the j-th pixel before illumination compensation, is the light brightness of the j-th pixel, k is the preset weight coefficient, is the average light intensity of the user's hand area.

[0011] Optionally, the aforementioned central control display content customization method based on image processing, before calculating the user's interest in the multiple locations according to the user's behavioral characteristics, further includes: collecting the user's voice and converting it into text for word segmentation; calculating the word frequency, document frequency, and context embedding vector of each word in the text; calculating the sentiment tendency of each word, wherein the sentiment tendency of the mth word ,in is the preset weight value, is the context embedding vector of the mth word, is the offset value; calculate the comprehensive score of each word ,in is the frequency of the mth word, N is the number of all documents in the preset document set, is the document frequency of the m-th word, indicating the number of documents in the document set that contain the m-th word; based on the comprehensive score of each word, keywords are selected from the text; and the keywords are used to supplement the behavioral characteristics of the user.

[0012] Optionally, the aforementioned central control display content customization method based on image processing obtains content related to the target location and displays it through the central control system of the vehicle, including: obtaining multiple contents related to the target location; calculating the information capacity of the multiple contents; setting the display area of ​​the multiple contents on the central control system according to the information content of the multiple contents; and displaying the multiple contents through the display area set on the central control system.

[0013] In the second aspect, the present application provides a central control display content customization system based on image processing, including: a route position module, which obtains the vehicle's driving route and the current position of the vehicle; a location prediction module, which predicts multiple locations that the vehicle will pass through based on the vehicle's driving route and current position; an image capture module, which captures an image of a user in the vehicle; a feature recognition module, which identifies the user's behavioral characteristics based on the user's image; an interest calculation module, which calculates the user's interest level in the multiple locations based on the user's behavioral characteristics; a location selection module, which selects a target location from the multiple locations based on the user's interest level in the multiple locations; and a display module, which obtains content related to the target location and displays it through the vehicle's central control system.

[0014] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0015] According to the technical solution of the present invention, the user's interests and intentions can be inferred based on the user's facial expressions, sight direction, gesture type and other behavioral characteristics, so as to provide a more personalized central control system display service to meet the user's personalized needs. For example, if the user shows interest in a certain scenic spot during driving (such as frequently looking at or pointing in a certain direction), the system can automatically display relevant information about the scenic spot, including maps, pictures, text descriptions, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 is a flow chart of a method for customizing central control display content based on image processing according to an embodiment of the present application;

[0018] Figure 2 is another flow chart of a method for customizing central control display content based on image processing according to an embodiment of the present application;

[0019] Figure 3 is another flow chart of a method for customizing central control display content based on image processing according to an embodiment of the present application;

[0020] Figure 4 A partial flow chart of a method for customizing central control display content based on image processing according to an embodiment of the present application;

[0021] Figure 5 The present invention is a block diagram of a central control display content customization system based on image processing according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Some embodiments of the present application are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0023] like Figure 1 As shown, an embodiment of the present application provides a method for customizing central control display content based on image processing, comprising:

[0024] Step S110, obtaining the vehicle's driving route and the vehicle's current position.

[0025] Step S120, predicting multiple locations that the vehicle will pass through based on the vehicle's driving route and current position.

[0026] In this embodiment, the current position and driving route of the vehicle can be updated in real time, and the predicted passing places can be dynamically adjusted accordingly. This enables the system to respond to changes in road conditions in a timely manner and provide the latest information.

[0027] Step S130 , capturing an image of the user in the vehicle.

[0028] Step S140, identifying the user's behavior characteristics based on the user's image.

[0029] Step S150, calculating the user's interest in multiple locations based on the user's behavior characteristics.

[0030] In this embodiment, by analyzing the user's behavioral characteristics in real time, the system can intelligently filter and display relevant information to reduce unnecessary information interference. This helps the driver concentrate and reduces the risk of distracted driving.

[0031] Step S160 , selecting a target location from the multiple locations based on the user's interest in the multiple locations.

[0032] Step S170, obtaining content related to the target location and displaying it through the vehicle's central control system.

[0033] According to the technical solution of this embodiment, the user's interests and intentions can be inferred based on the user's facial expressions, sight direction, gesture type and other behavioral characteristics. In this way, the system can provide more personalized information and services to meet the specific needs of users. For example, if the user shows interest in a certain scenic spot during driving (such as frequently looking at or pointing in a certain direction), the system can automatically display relevant information about the scenic spot, including maps, pictures, text descriptions, etc.

[0034] An embodiment of the present application further provides a method for customizing central control display content based on image processing. Compared with the above-mentioned embodiment, the method for customizing central control display content based on image processing in this embodiment includes: step S140:

[0035] (1) When there are multiple users in a vehicle, the identities of the multiple users are identified from an image captured containing the multiple users.

[0036] (2) Analyze the relationship between multiple users based on their identities.

[0037] (3) Select one user from multiple users based on the relationship between them and analyze the behavioral characteristics of the selected user.

[0038] According to the technical solution of this embodiment, the system can flexibly adjust the displayed content according to the relationship between multiple users and the selected primary user. This makes the information display more targeted and reduces unnecessary interference. For example, if there are children in the car, the system can display content suitable for children, such as cartoons or educational videos; if there are elderly people in the car, it can display easy-to-read large font information.

[0039] like Figure 2 As shown, an embodiment of the present application further provides a method for customizing central control display content based on image processing. Compared with the above-mentioned embodiment, the method for customizing central control display content based on image processing in this embodiment includes: step S140:

[0040] Step S210, identifying the user's head from the user's image.

[0041] Step S220, detecting the center position of the user's eyeball from the user's head position and pupil center , and analyze the horizontal and vertical deflection angles of the user's head .

[0042] Specifically include:

[0043] (1) Detect the user's facial key points from the user's head, where the coordinates of the i-th facial key point are .

[0044] (2) Establishing the projection error function ,in, The function is the projection function from three-dimensional space to two-dimensional plane. is the i-th facial key point on the preset standard facial model, , They are rotation matrix and translation matrix respectively, and solve the projection error function Rotate the matrix when taking the minimum value , translation matrix The value of .

[0045] (3) Rotation matrix Decompose to get the horizontal and vertical deflection angles of the user's head .

[0046] In this embodiment, by accurately identifying the head position and facial key points from the user's image and combining the three-dimensional to two-dimensional projection error function, the user's line of sight direction can be calculated with high precision.

[0047] Step S230: Calculate the user's sight direction , as the behavioral characteristics of users.

[0048] In this embodiment, high-precision line of sight direction detection is achieved by accurately identifying the user's head position, detecting the center position of the eyeball and the center position of the pupil, and analyzing the deflection angles of the head in the horizontal and vertical directions to calculate the user's line of sight direction.

[0049] According to the technical solution of this embodiment, the system can infer the user's interest in different content based on the user's line of sight, and make personalized content recommendations accordingly. For example, if the system detects that the user frequently looks at a certain attraction or restaurant, it can automatically push relevant detailed information, such as business hours, menus, reviews, etc.

[0050] like Figure 3 As shown, an embodiment of the present application further provides a method for customizing central control display content based on image processing. Compared with the above-mentioned embodiment, the method for customizing central control display content based on image processing in this embodiment includes: step S140:

[0051] Step S310, identifying the user's hand area from the user's image.

[0052] Step S320: Perform illumination compensation on the user's hand area according to a preset method.

[0053] In this embodiment, by performing illumination compensation on the user's hand area, the system can effectively reduce the impact of illumination changes on gesture recognition. This makes gesture recognition more accurate and stable. Under different lighting conditions (such as strong light, weak light or uneven lighting), illumination compensation can ensure the image quality of the hand area, thereby improving the robustness of gesture recognition.

[0054] Step S330 , identifying the user's gestures based on the user's hand area after illumination compensation as the user's behavior features.

[0055] In this embodiment, based on the user's gestures, the system can infer the user's interests and needs and make personalized content recommendations accordingly. For example, if the system detects that the user frequently gestures to a certain location, relevant information about the location, such as maps, pictures, text descriptions, etc., can be automatically displayed.

[0056] Specifically include:

[0057] (1) Detect the light intensity of each pixel in the user's hand area.

[0058] (2) Calculate the average light intensity of the user’s hand area.

[0059] (3) Perform illumination compensation on each pixel in the user's hand area, where the pixel value after illumination compensation of the jth pixel is ,in is the preset target light brightness, is the pixel value of the jth pixel before illumination compensation, is the light brightness of the jth pixel, k is the preset weight coefficient, is the average light intensity of the user's hand area.

[0060] According to the technical solution of this embodiment, high-precision gesture recognition is achieved by identifying the user's hand area, performing illumination compensation, and identifying the user's gestures based on the hand area after illumination compensation. This method significantly improves user experience, driving safety, and the intelligence level of the system, providing users with a more natural and convenient interaction method.

[0061] like Figure 4 As shown, an embodiment of the present application further provides a central control display content customization method based on image processing. Compared with the aforementioned embodiment, the central control display content customization method based on image processing in this embodiment, before step S150, further includes:

[0062] Step S410: collect the user's voice and convert it into text for word segmentation.

[0063] Step S420, calculating the word frequency, document frequency, and context embedding vector of each word in the text.

[0064] Step S430, calculating the sentiment tendency of each word, where the sentiment tendency of the mth word is ,in is the preset weight value, is the context embedding vector of the mth word, is the offset value.

[0065] In this embodiment, by calculating the sentiment tendency of each word, the system can better understand the user's emotional state and select keywords that match the user's emotions.

[0066] Step S440, calculating the comprehensive score of each word ,in is the frequency of the mth word, N is the number of all documents in the preset document set, is the document frequency of the mth word, which indicates the number of documents in the document collection that contain the mth word.

[0067] Step S450, selecting keywords from the text according to the comprehensive score of each word.

[0068] Step S460: Use keywords to supplement the user's behavior characteristics.

[0069] According to the technical solution of this embodiment, by combining voice and image information, the system can achieve the fusion of multimodal information, thereby more comprehensively understanding the user's behavioral characteristics and intentions. For example, by combining the user's facial expressions, gestures and voice content, the system can more accurately infer the user's interests and needs.

[0070] An embodiment of the present application further provides a method for customizing central control display content based on image processing. Compared with the above-mentioned embodiment, the method for customizing central control display content based on image processing in this embodiment includes step S170:

[0071] (1) Obtain multiple contents related to the target location.

[0072] (2) Calculate the information capacity of multiple contents.

[0073] (3) According to the information content of multiple contents, set the display area of ​​multiple contents on the central control system.

[0074] In this embodiment, by calculating the information capacity of each content, the system can reasonably arrange and layout the display area to ensure that important information is fully displayed while avoiding information overload. For example, for content with a large amount of information (such as detailed text descriptions), the system can allocate a larger display area; while for content with a small amount of information (such as icons or brief prompts), a smaller area can be allocated.

[0075] (4) Display multiple contents through the display area set on the central control system.

[0076] According to the technical solution of this embodiment, by obtaining multiple contents related to the target location, calculating the information capacity, setting the display area, and displaying these contents on the central control system, the optimization of information display and the improvement of user experience are achieved. This method significantly improves the efficiency of information transmission and driving safety, and provides users with a more natural and convenient interaction method.

[0077] like Figure 5 As shown, an embodiment of the present application provides a central control display content customization system based on image processing, comprising:

[0078] The route location module 510 obtains the vehicle's driving route and the vehicle's current location.

[0079] The location prediction module 520 predicts a plurality of locations that the vehicle will pass through based on the vehicle's driving route and current location.

[0080] In this embodiment, the current position and driving route of the vehicle can be updated in real time, and the predicted passing places can be dynamically adjusted accordingly. This enables the system to respond to changes in road conditions in a timely manner and provide the latest information.

[0081] The image capturing module 530 captures an image of a user in the vehicle.

[0082] The feature recognition module 540 recognizes the user's behavior features based on the user's image.

[0083] The interest calculation module 550 calculates the user's interest in multiple locations based on the user's behavioral characteristics.

[0084] In this embodiment, by analyzing the user's behavioral characteristics in real time, the system can intelligently filter and display relevant information to reduce unnecessary information interference. This helps the driver concentrate and reduces the risk of distracted driving.

[0085] The location selection module 560 selects a target location from the multiple locations according to the user's interest in the multiple locations.

[0086] The display module 570 obtains the content related to the target location and displays it through the central control system of the vehicle.

[0087] According to the technical solution of this embodiment, the user's interests and intentions can be inferred based on the user's facial expressions, sight direction, gesture type and other behavioral characteristics. In this way, the system can provide more personalized information and services to meet the specific needs of users. For example, if the user shows interest in a certain scenic spot during driving (such as frequently looking at or pointing in a certain direction), the system can automatically display relevant information about the scenic spot, including maps, pictures, text descriptions, etc.

[0088] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0089] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0090] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0091] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0092] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for customizing central control display content based on image processing, characterized in that: include: Obtaining the driving route of the vehicle and the current position of the vehicle; Predicting a plurality of locations that the vehicle will pass through based on the driving route and current position of the vehicle; capturing an image of a user within the vehicle; identifying behavioral characteristics of the user according to the image of the user; Calculating the user's interest in the multiple locations based on the user's behavioral characteristics; selecting a target location from the multiple locations according to the user's interest in the multiple locations; Obtaining content related to the target location and displaying it through the vehicle's central control system, Before calculating the user's interest in the plurality of locations according to the user's behavior characteristics, the method further includes: Collecting the user's voice and converting it into text for word segmentation; Calculate the word frequency, document frequency, and context embedding vector of each word in the text; Calculate the sentiment tendency of each word, where the sentiment tendency of the mth word ,in, is the preset weight value, is the context embedding vector of the mth word, is the offset value; Calculate the comprehensive score of each word ,in, is the frequency of the mth word, N is the number of all documents in the preset document set, is the document frequency of the m-th word, indicating the number of documents in the document set that contain the m-th word; Selecting keywords from the text according to the comprehensive score of each word; The keywords are used to supplement the behavioral characteristics of the user.

2. The method for customizing central control display content based on image processing according to claim 1, characterized in that: Identifying the behavior characteristics of the user according to the image of the user includes: When there are multiple users in the vehicle, identifying the identities of the multiple users from the captured images containing the multiple users; Analyzing the relationship between the multiple users according to the identities of the multiple users; According to the relationship between the multiple users, one user is selected from the multiple users, and the behavior characteristics of the selected user are analyzed.

3. The method for customizing central control display content based on image processing according to claim 1, characterized in that: Identifying the behavior characteristics of the user according to the image of the user includes: From the image of the user, identifying a head of the user; Detect the center position of the user's eyeball from the user's head position and pupil center , and analyzing the deflection angles of the user's head in the horizontal and vertical directions ; Calculate the user's line of sight , as the behavioral characteristics of the user.

4. The method for customizing central control display content based on image processing according to claim 3, characterized in that: Analyze the deflection angle of the user's head in the horizontal and vertical directions ,include: From the user's head, detect the user's facial key points, where the coordinates of the i-th facial key point are ; Establishing the projection error function ,in, The function is the projection function from three-dimensional space to two-dimensional plane. is the i-th facial key point on the preset standard facial model, , They are rotation matrix and translation matrix respectively, and solve the projection error function The rotation matrix is ​​minimized , the translation matrix The value of For the rotation matrix Decompose to obtain the deflection angle of the user's head in the horizontal and vertical directions .

5. The method for customizing central control display content based on image processing according to claim 1, characterized in that: Identifying the behavior characteristics of the user according to the image of the user includes: identifying a hand region of the user from the image of the user; Performing illumination compensation on the hand area of ​​the user in a preset manner; According to the hand area of ​​the user after illumination compensation, the user's gesture action is identified as the user's behavior feature.

6. The method for customizing central control display content based on image processing according to claim 5, characterized in that: Performing illumination compensation on the hand area of ​​the user in a preset manner includes: Detecting the light intensity of each pixel in the hand area of ​​the user; Calculating the average light intensity of the user's hand area; Perform illumination compensation on each pixel in the user's hand area, where the pixel value of the jth pixel after illumination compensation is ,in, is the preset target light brightness, is the pixel value of the j-th pixel before illumination compensation, is the light brightness of the j-th pixel, k is the preset weight coefficient, is the average light intensity of the user's hand area.

7. The method for customizing central control display content based on image processing according to claim 1, characterized in that: Obtaining content related to the target location and displaying it through the vehicle's central control system, including: Acquire multiple contents related to the target location; Calculating the information capacity of the plurality of contents; According to the information content of the multiple contents, setting the display area of ​​the multiple contents on the central control system; The multiple contents are displayed through a display area set on the central control system.

8. The central control display content customization system based on image processing is characterized by: include: A route location module, which obtains the driving route of the vehicle and the current location of the vehicle; A location prediction module, which predicts a plurality of locations that the vehicle will pass through based on the driving route and current location of the vehicle; An image capturing module, configured to capture an image of a user in the vehicle; A feature recognition module, which recognizes the user's behavior features based on the user's image; an interest calculation module, calculating the user's interest level in the plurality of locations according to the user's behavior characteristics; a location selection module, selecting a target location from the multiple locations according to the user's interest in the multiple locations; The display module obtains the content related to the target location and displays it through the vehicle's central control system. Before calculating the user's interest in the multiple locations according to the user's behavior characteristics, the central control display content customization system based on image processing further executes: Collecting the user's voice and converting it into text for word segmentation; Calculate the word frequency, document frequency, and context embedding vector of each word in the text; Calculate the sentiment tendency of each word, where the sentiment tendency of the mth word ,in, is the preset weight value, is the context embedding vector of the mth word, is the offset value; Calculate the comprehensive score of each word ,in, is the frequency of the mth word, N is the number of all documents in the preset document set, is the document frequency of the m-th word, indicating the number of documents in the document set that contain the m-th word; Selecting keywords from the text according to the comprehensive score of each word; The keywords are used to supplement the behavioral characteristics of the user.

Images