Vehicle-mounted teaching method and system, computer equipment and readable storage medium

Abstract

The invention provides a vehicle-mounted teaching method and system, computer equipment and a readable storage medium, and the method comprises the steps: receiving a voice instruction of a child in a vehicle through a vehicle-mounted microphone, and collecting a child image of the child in the vehicle through a vehicle-mounted camera; performing gesture recognition on the child in the vehicle based on the child image through the vehicle-mounted processor, and judging whether a gesture recognition result is a pointing gesture or not; if the gesture recognition result is a pointing gesture, determining a target detection area through a vehicle-mounted processor according to the gesture recognition result, and collecting an area image of the target detection area through a vehicle-mounted camera; image matching information is determined based on the area image and the voice instruction through the vehicle-mounted processor, and voice broadcasting is conducted on the image matching information to the child in the vehicle through the vehicle-mounted loudspeaker. By adopting the method, the vehicle-mounted teaching of children in the vehicle environment is realized, and meanwhile, the driving safety of the vehicle in the vehicle-mounted teaching process is ensured.

Description

Technical Field

[0001] The present invention relates to the field of vehicle intelligent management technology, and in particular to an in-vehicle teaching method, system, computer equipment and readable storage medium. Background Art

[0002] With the rapid development of new energy vehicle technology, car cockpits are also evolving towards intelligence. Users' demands for cars have been upgraded from traditional functional demands to enjoyment demands, and they are more concerned about the intelligence, multi-functions and comfort of vehicles. In existing technologies, more and more smart cockpits are being put into use, providing users with ways to interact with vehicle systems through voice.

[0003] With the rapid development of new energy vehicle technology, car cabins are also evolving towards intelligence. Users' demand for cars has been upgraded to enjoyment needs based on traditional functional needs, and there are more and more car products that focus on the needs of family users.

[0004] However, the research found that the current smart cockpits are lacking in child companionship and teaching functions. Most manufacturers have not considered the situation where children read picture books, textbooks, comic books, etc. in the cockpit. When children read and study in the cockpit, they usually can only ask their parents for help when they encounter unclear questions. When parents are driving, they cannot answer questions for children or teach them. In addition, because the driver is disturbed and influenced by children's questions during driving, it will cause distraction and affect driving safety. Summary of the invention

[0005] In view of this, an object of the present invention is to provide an in-vehicle teaching method, system, computer device and readable storage medium to realize in-vehicle teaching for children in an in-vehicle environment while ensuring the driving safety of the vehicle during the in-vehicle teaching process.

[0006] In a first aspect, an embodiment of the present application provides an in-vehicle teaching method, which is applied to an in-vehicle teaching system, wherein the system includes an in-vehicle processor, an in-vehicle camera, an in-vehicle microphone, and an in-vehicle speaker, and the method includes:

[0007] receiving a voice command of a child in the car through the onboard microphone, and collecting an image of the child in the car through the onboard camera;

[0008] performing gesture recognition of the child in the vehicle based on the child image by the vehicle-mounted processor, and determining whether the gesture recognition result is a pointing gesture;

[0009] If the gesture recognition result is a pointing gesture, determining a target detection area according to the gesture recognition result by the vehicle-mounted processor, and collecting a regional image of the target detection area by the vehicle-mounted camera;

[0010] The vehicle-mounted processor determines image matching information based on the area image and the voice command, and the image matching information is voice broadcasted to the child in the vehicle through the vehicle-mounted speaker.

[0011] Optionally, the method comprises:

[0012] If the gesture recognition result is not a pointing gesture, the target response information is determined by the vehicle-mounted processor based on the voice command, and the target response information is voice broadcasted to the child in the vehicle through the vehicle-mounted speaker.

[0013] Optionally, the performing gesture recognition on the child in the vehicle based on the child image includes:

[0014] Performing hand detection on the child image based on a hand detection model to obtain a hand detection frame;

[0015] A hand region image obtained by cropping the child image based on the hand detection frame;

[0016] The hand area image is classified and identified based on a gesture classifier to perform gesture recognition on the child in the car.

[0017] Optionally, determining a target detection area according to the gesture recognition result includes:

[0018] Segmenting the hand region in the child image, and performing threshold processing on the child image based on the segmentation result to determine the hand region;

[0019] A detection area starting point is determined according to an area boundary of the hand area, and the target detection area is determined according to the detection area starting point and area selection parameters.

[0020] Optionally, before performing gesture recognition on the child in the vehicle based on the child image by the vehicle-mounted processor, the method further includes:

[0021] The child image is subjected to image grayscale processing by the vehicle-mounted processor.

[0022] Optionally, the determining the image matching information based on the regional image and the voice command includes:

[0023] Based on the image recognition model, feature extraction and classification processing are performed on the regional image in sequence to obtain a target classification result of the regional image;

[0024] Performing semantic recognition on the voice command based on a semantic recognition model to obtain target semantic information contained in the voice command;

[0025] Image matching information matching the target semantic information and the target classification result is determined based on the speech image matching model.

[0026] Optionally, the system further includes a vehicle-mounted display, and the method further includes:

[0027] The image matching information is displayed to the child in the vehicle via the vehicle-mounted display.

[0028] In a second aspect, an embodiment of the present application provides an in-vehicle teaching system, the system comprising an in-vehicle processor, an in-vehicle camera, an in-vehicle microphone, and an in-vehicle speaker;

[0029] The vehicle-mounted microphone is used to receive voice commands from children in the vehicle;

[0030] The vehicle-mounted camera is used to collect images of children in the vehicle;

[0031] The vehicle-mounted processor is used to perform gesture recognition on the child in the vehicle based on the child image, and determine whether the gesture recognition result is a pointing gesture; if the gesture recognition result is a pointing gesture, determine a target detection area according to the gesture recognition result;

[0032] The vehicle-mounted camera is used to collect the regional image of the target detection area;

[0033] The vehicle-mounted processor is used to determine image matching information based on the area image and the voice command;

[0034] The vehicle-mounted speaker is used to voice broadcast the image matching information to the children in the vehicle.

[0035] Optionally, the vehicle-mounted processor is used to determine target reply information based on the voice instruction if the gesture recognition result is not a pointing gesture;

[0036] The vehicle-mounted speaker is used to voice broadcast the target response information to the children in the vehicle.

[0037] Optionally, the performing gesture recognition on the child in the vehicle based on the child image includes:

[0038] Performing hand detection on the child image based on a hand detection model to obtain a hand detection frame;

[0039] A hand region image obtained by cropping the child image based on the hand detection frame;

[0040] The hand area image is classified and identified based on a gesture classifier to perform gesture recognition on the child in the car.

[0041] Optionally, determining a target detection area according to the gesture recognition result includes:

[0042] Segmenting the hand region in the child image, and performing threshold processing on the child image based on the segmentation result to determine the hand region;

[0043] A detection area starting point is determined according to an area boundary of the hand area, and the target detection area is determined according to the detection area starting point and area selection parameters.

[0044] Optionally, the vehicle-mounted processor is used to perform grayscale processing on the child image before performing gesture recognition on the child in the vehicle based on the child image through the vehicle-mounted processor.

[0045] Optionally, the determining the image matching information based on the regional image and the voice command includes:

[0046] Based on the image recognition model, feature extraction and classification processing are performed on the regional image in sequence to obtain a target classification result of the regional image;

[0047] Performing semantic recognition on the voice command based on a semantic recognition model to obtain target semantic information contained in the voice command;

[0048] Image matching information matching the target semantic information and the target classification result is determined based on the speech image matching model.

[0049] Optionally, the system further comprises a vehicle-mounted display, and the vehicle-mounted display is used to display the image matching information to the child in the vehicle.

[0050] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the in-vehicle teaching method described in any optional implementation manner of the first aspect are performed.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the in-vehicle teaching method described in any optional implementation manner in the above-mentioned first aspect are executed.

[0052] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0053] The present application collects and recognizes children's voice commands and pointing gestures, matches the area pointed by the pointing gesture with the voice command, determines the image matching information in the area pointed by the gesture for answering the children's voice commands, and voice broadcasts them to the children. This enables in-car teaching for children while ensuring the driving safety of the vehicle during the in-car teaching process.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A flow chart of an in-vehicle teaching method provided by the first embodiment of the present invention is shown;

[0057] Figure 2 A flowchart of a gesture recognition method provided by Embodiment 1 of the present invention is shown;

[0058] Figure 3 A flow chart of a method for determining a target detection area provided by Embodiment 1 of the present invention is shown;

[0059] Figure 4 A flowchart of a method for determining image matching information provided by Embodiment 1 of the present invention is shown;

[0060] Figure 5 A schematic diagram showing the structure of a vehicle-mounted teaching system provided by a second embodiment of the present invention is shown;

[0061] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0063] Embodiment 1

[0064] To facilitate understanding of this application, Figure 1 The flowchart of an in-vehicle teaching method provided by the first embodiment of the present invention is shown to describe the contents of the first embodiment of the present application in detail.

[0065] See also Figure 1 As shown, Figure 1 The flowchart of a vehicle-mounted teaching method provided in the first embodiment of the present invention is shown, wherein the method comprises steps S101 to S104:

[0066] S101: receiving a voice command of a child in the car through the car microphone, and collecting an image of the child in the car through the car camera.

[0067] S102: Performing gesture recognition on the child in the vehicle based on the child image by the vehicle-mounted processor to determine whether the gesture recognition result is a pointing gesture.

[0068] S103: If the gesture recognition result is a pointing gesture, determining a target detection area according to the gesture recognition result by the vehicle-mounted processor, and collecting a regional image of the target detection area by the vehicle-mounted camera.

[0069] S104: Determine image matching information based on the area image and the voice command through the vehicle-mounted processor, and voice broadcast the image matching information to the child in the vehicle through the vehicle-mounted speaker.

[0070] Specifically, the voice wake-up signal of the child in the car is received through the on-board microphone to wake up the teaching system, and then the voice command of the child in the car is received through the on-board microphone. The child image of the child in the car is collected through the on-board camera, and the gesture in the child image is recognized to determine whether it is a pointing gesture. When the child's gesture is recognized as a pointing gesture, the pointing area above the finger is identified and located as the target detection area. The camera above the child is called to collect the regional image of the target detection area, and the voice command and the regional image are input into the multimodal large model together. After receiving the voice command and the regional image, the multimodal large model first pre-processes the regional image, then performs feature segmentation and extraction, and then performs voice recognition on the voice command to obtain one-dimensional text information, and then performs semantic recognition, identifies the problem corresponding to the semantic recognition and corresponds to the image features in the regional image to obtain image matching information, and finally uses the image matching information as the matching result for voice broadcasting through the on-board speaker.

[0071] In an alternative embodiment, the method comprises:

[0072] If the gesture recognition result is not a pointing gesture, the target response information is determined by the vehicle-mounted processor based on the voice command, and the target response information is voice broadcasted to the child in the vehicle through the vehicle-mounted speaker.

[0073] Specifically, if the gesture recognition result is not a pointing gesture, the voice command is directly transmitted to the large model to determine the target reply information, and the target reply information is voice broadcasted.

[0074] For example, when the voice command is "What is the temperature in degrees Celsius today?", the "What is the temperature in degrees Celsius today" is transmitted to the large model to determine the target response information as "26 degrees Celsius", and "26 degrees Celsius" is voice broadcast to the children in the car through the car's speakers.

[0075] In an alternative embodiment, see Figure 2 As shown, Figure 2 A flowchart of a gesture recognition method provided by the first embodiment of the present invention is shown, wherein the gesture recognition of the child in the car based on the child image includes steps S201 to S203:

[0076] S201: Perform hand detection on the child image based on a hand detection model to obtain a hand detection frame.

[0077] S202: obtaining a hand region image by cropping the child image based on the hand detection frame.

[0078] S203: Classify and identify the hand area image based on a gesture classifier to perform gesture recognition on the child in the car.

[0079] Specifically, the specific implementation of gesture recognition is to first train a general hand detection model based on the YOLO algorithm (without distinguishing gestures, only detecting the hand frame), then input the child image into the hand detection model to obtain the hand detection frame, crop the area contained in the hand detection frame to obtain the hand area image, and then train a gesture classifier, and input the hand area image into the gesture classifier to complete the classification and recognition of pointing gestures.

[0080] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flow chart of a method for determining a target detection area provided by the first embodiment of the present invention is shown, wherein the method for determining the target detection area according to the gesture recognition result includes steps S301 to S302:

[0081] S301: Segment the hand region in the child image, and perform threshold processing on the child image based on the segmentation result to determine the hand region.

[0082] S302: Determine a detection area starting point according to the area boundary of the hand area, and determine the target detection area according to the detection area starting point and area selection parameters.

[0083] Specifically, after the child's gesture is identified as a pointing gesture, the pointing area above the finger is identified and located. Specifically, after the hand area is segmented from the real-time video sequence obtained by the camera, the image is thresholded so that only the hand area is visible, and all other unnecessary areas are painted black. After thresholding, the hand boundary contour is found in the result image. It is assumed that the contour with the largest area in the image is the hand, and the contour line is the contour or boundary of the object in the image. Finally, the convex hull (contour coverage) of the segmented hand area is found, and the most extreme point in the convex hull (extreme top, i.e., the fingertip of the finger that realizes the pointing function in the pointing gesture) is determined as the starting point of the detection area. The pointing direction of the pointing gesture is used as the area expansion direction, and a rectangular area with a preset length and width is expanded from the starting point of the detection area as the target detection area.

[0084] In an optional embodiment, before performing gesture recognition of the child in the vehicle based on the child image by the vehicle-mounted processor, the method further includes:

[0085] The child image is subjected to image grayscale processing by the vehicle-mounted processor.

[0086] Specifically, if the model directly processes color images, it often needs to process the three channels of R, G, and B in sequence, which will take a lot of time. Therefore, in order to improve the processing speed of the entire system, it is necessary to grayscale the children's images to reduce the amount of data that needs to be processed.

[0087] In an alternative embodiment, see Figure 4 As shown, Figure 4 A flowchart of a method for determining image matching information provided by the first embodiment of the present invention is shown, wherein the method for determining image matching information based on the regional image and the voice command comprises steps S401 to S403:

[0088] S401: performing feature extraction and classification processing on the regional image in sequence based on an image recognition model to obtain a target classification result of the regional image.

[0089] S402: Performing semantic recognition on the voice instruction based on a semantic recognition model to obtain target semantic information contained in the voice instruction.

[0090] S403: Determine image matching information that matches the target classification result for the target semantic information based on the speech image matching model.

[0091] Specifically, the image recognition model, the semantic recognition model and the voice image matching model jointly construct a multimodal large model. The voice command and the regional image are input into the multimodal large model. After receiving the voice command and the regional image, the multimodal large model uses the image recognition model to extract features and classify the regional image to obtain the target classification result of the regional image. At the same time, the semantic recognition model is used to perform semantic recognition on the voice command to obtain the target semantic information contained in the voice command. At the same time, the voice image matching model is used to determine the image matching information that matches the target classification result for the target semantic information, that is, to identify the voice command and correspond to the image features, and finally output the image matching information.

[0092] The multimodal large model here chooses to call Chat-gpt4's "gpt-4-vision-preview" API to implement image recognition. This technology uses pre-trained image recognition models (ResNet, Inception, etc.) to classify and extract features from images, and then inputs the classification results and feature vectors into the GPT model to allow the model to generate text in the image that is related to the content of the voice command.

[0093] For example, when a child is studying in a car, he sees a beautiful building in a book, but he doesn't know what it is, so he starts to ask in voice: "What is this building?" and points his finger at the picture of the building in the book. At this time, the child teaching system based on the multimodal large model starts to work. By understanding the semantics, it determines that the child needs to identify the picture information pointed by the finger. After determining the target area, the camera above the child is called to take the image. After preprocessing, the image and the question are input into the large model. The large model performs image segmentation and feature extraction based on the keyword "building" in the sentence, and gives feedback after obtaining the result. At this time, the voice broadcasts: "Hello, child, this building is the Sun Tower."

[0094] In an optional embodiment, the method further comprises:

[0095] The image matching information is displayed to the child in the vehicle via the vehicle-mounted display.

[0096] Specifically, in addition to voice broadcasting, the image matching information can also be displayed to children in the car in the form of text.

[0097] Embodiment 2

[0098] See also Figure 5 As shown, Figure 5 The structure diagram of a vehicle-mounted teaching system provided by the second embodiment of the present invention is shown, wherein the system includes a vehicle-mounted processor 501, a vehicle-mounted camera 502, a vehicle-mounted microphone 503 and a vehicle-mounted speaker 504;

[0099] The vehicle-mounted microphone is used to receive voice commands from children in the vehicle;

[0100] The vehicle-mounted camera is used to collect images of children in the vehicle;

[0101] The vehicle-mounted processor is used to perform gesture recognition on the child in the vehicle based on the child image, and determine whether the gesture recognition result is a pointing gesture; if the gesture recognition result is a pointing gesture, determine a target detection area according to the gesture recognition result;

[0102] The vehicle-mounted camera is used to collect the regional image of the target detection area;

[0103] The vehicle-mounted processor is used to determine image matching information based on the area image and the voice command;

[0104] The vehicle-mounted speaker is used to voice broadcast the image matching information to the children in the vehicle.

[0105] In an optional embodiment, the vehicle-mounted processor is used to determine target reply information based on the voice instruction if the gesture recognition result is not a pointing gesture;

[0106] The vehicle-mounted speaker is used to voice broadcast the target response information to the children in the vehicle.

[0107] In an optional implementation, the performing gesture recognition on the child in the vehicle based on the child image includes:

[0108] Performing hand detection on the child image based on a hand detection model to obtain a hand detection frame;

[0109] A hand region image obtained by cropping the child image based on the hand detection frame;

[0110] The hand area image is classified and identified based on a gesture classifier to perform gesture recognition on the child in the car.

[0111] In an optional implementation, determining the target detection area according to the gesture recognition result includes:

[0112] Segmenting the hand region in the child image, and performing threshold processing on the child image based on the segmentation result to determine the hand region;

[0113] A detection area starting point is determined according to an area boundary of the hand area, and the target detection area is determined according to the detection area starting point and area selection parameters.

[0114] In an optional embodiment, the vehicle-mounted processor is used to perform grayscale processing on the child image before performing gesture recognition on the child in the vehicle based on the child image by the vehicle-mounted processor.

[0115] In an optional implementation, the determining of image matching information based on the regional image and the voice command includes:

[0116] Based on the image recognition model, feature extraction and classification processing are performed on the regional image in sequence to obtain a target classification result of the regional image;

[0117] Performing semantic recognition on the voice command based on a semantic recognition model to obtain target semantic information contained in the voice command;

[0118] Image matching information matching the target semantic information and the target classification result is determined based on the speech image matching model.

[0119] In an optional embodiment, the system further comprises a vehicle-mounted display, and the vehicle-mounted display is used to display the image matching information to the child in the vehicle.

[0120] Embodiment 3

[0121] Based on the same application concept, see Figure 6 As shown, Figure 6 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Figure 6 As shown, a computer device 600 provided in Embodiment 3 of the present application includes:

[0122] A processor 601, a memory 602 and a bus 603, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the computer device 600 is running, the processor 601 communicates with the memory 602 via the bus 603. When the processor 601 is running, the machine-readable instructions execute the steps of the in-vehicle teaching method shown in the above-mentioned embodiment 1.

[0123] Embodiment 4

[0124] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the in-vehicle teaching method described in any one of the above embodiments are executed.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0126] The computer program product for in-vehicle teaching provided in an embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0127] The in-vehicle teaching system provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The system provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the system embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0128] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0131] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0132] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0133] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A vehicle-based teaching method, It is characterized in that Applied to an in-vehicle teaching system, the system includes an in-vehicle processor, an in-vehicle camera, an in-vehicle microphone and an in-vehicle speaker, and the method includes: receiving a voice command of a child in the car through the onboard microphone, and collecting an image of the child in the car through the onboard camera; performing gesture recognition of the child in the vehicle based on the child image by the vehicle-mounted processor, and determining whether the gesture recognition result is a pointing gesture; If the gesture recognition result is a pointing gesture, determining a target detection area according to the gesture recognition result by the vehicle-mounted processor, and collecting a regional image of the target detection area by the vehicle-mounted camera; The vehicle-mounted processor determines image matching information based on the area image and the voice command, and the image matching information is voice broadcasted to the child in the vehicle through the vehicle-mounted speaker.

2. The method according to claim 1, It is characterized in that The method comprises: If the gesture recognition result is not a pointing gesture, the target response information is determined by the vehicle-mounted processor based on the voice command, and the target response information is voice broadcasted to the child in the vehicle through the vehicle-mounted speaker.

3. The method according to claim 1, It is characterized in that The performing gesture recognition on the child in the vehicle based on the child image includes: Performing hand detection on the child image based on a hand detection model to obtain a hand detection frame; A hand region image obtained by cropping the child image based on the hand detection frame; The hand area image is classified and identified based on a gesture classifier to perform gesture recognition on the child in the car.

4. The method according to claim 1, It is characterized in that The determining of the target detection area according to the gesture recognition result includes: Segmenting the hand region in the child image, and performing threshold processing on the child image based on the segmentation result to determine the hand region; A detection area starting point is determined according to an area boundary of the hand area, and the target detection area is determined according to the detection area starting point and area selection parameters.

5. The method according to claim 1, It is characterized in that Before performing gesture recognition on the child in the vehicle based on the child image by the vehicle-mounted processor, the method further includes: The child image is subjected to image grayscale processing by the vehicle-mounted processor.

6. The method according to claim 1, It is characterized in that The determining of image matching information based on the regional image and the voice command includes: Based on the image recognition model, feature extraction and classification processing are performed on the regional image in sequence to obtain a target classification result of the regional image; Performing semantic recognition on the voice command based on a semantic recognition model to obtain target semantic information contained in the voice command; Image matching information matching the target semantic information and the target classification result is determined based on the speech image matching model.

7. The method according to claim 1, It is characterized in that The system further includes an in-vehicle display, and the method further includes: The image matching information is displayed to the child in the vehicle via the vehicle-mounted display.

8. A vehicle-mounted teaching system, It is characterized in that The system includes an on-board processor, an on-board camera, an on-board microphone and an on-board speaker; The vehicle-mounted microphone is used to receive voice commands from children in the vehicle; The vehicle-mounted camera is used to collect images of children in the vehicle; The vehicle-mounted processor is used to perform gesture recognition on the child in the vehicle based on the child image, and determine whether the gesture recognition result is a pointing gesture; If the gesture recognition result is a pointing gesture, determining a target detection area according to the gesture recognition result; The vehicle-mounted camera is used to collect the regional image of the target detection area; The vehicle-mounted processor is used to determine image matching information based on the area image and the voice command; The vehicle-mounted speaker is used to voice broadcast the image matching information to the children in the vehicle.

9. A computer device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the in-vehicle teaching method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the in-vehicle teaching method as claimed in any one of claims 1 to 7.

Images