Image recognition method and related device

CN118922868APending Publication Date: 2024-11-08YINWANG INTELLIGENT TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380027798.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-02-17
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing technology is difficult to accurately identify complex traffic signs, affecting the safety and reliability of autonomous driving systems.

Method used

Using the image recognition method, the first neural network and the second neural network are used to identify traffic signs in the road environment, combined with the intersection-union ratio (IOU) for matching, to determine the components of the composite sign and its meaning, save computing resources, and improve recognition efficiency. .

Benefits of technology

It achieves accurate recognition of complex traffic signs, improves the safety and reliability of the autonomous driving system, reduces the occupation of computing resources, and improves recognition efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922868A_ABST
    Figure CN118922868A_ABST
Patent Text Reader

Abstract

The invention provides an image recognition method and a related device, and relates to the technical field of vehicles, and the method comprises the steps: obtaining a first image; the first image is recognized based on a first neural network, a first recognition result is obtained, and the first recognition result is used for indicating that the first image comprises N first marks; the first image is recognized based on a second neural network, a second recognition result is obtained, the second recognition result is used for indicating that the first image comprises a second mark, and the second mark is a composite mark; respectively matching the first marks with the second marks to determine that the second marks comprise M first marks in the N first marks; and fusing the mark meanings corresponding to the M first marks to determine the mark meaning corresponding to the second mark. Based on the scheme, conventional traffic signs in a road environment can be accurately identified, and complex composite traffic signs can be formed.
Need to check novelty before this filing date? Find Prior Art

Description

Image recognition method and related device Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to an image recognition method and related devices. Background Art

[0002] Traffic sign detection technology is a key component of autonomous driving, helping autonomous systems better understand road traffic information, such as speed limits, road restrictions, and road condition warnings. However, the inability to accurately identify traffic signs in the road environment, especially those with complex structures, can significantly reduce the safety and reliability of autonomous driving systems.

[0003] Therefore, how to accurately identify traffic signs in road environments, especially those with complex structures, to improve the safety and reliability of autonomous driving systems is a technical problem that needs to be solved urgently.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide an image recognition method and related devices that can accurately identify conventional traffic signs in a road environment and more complex composite traffic signs, thereby helping to improve the safety and reliability of an autonomous driving system or an assisted driving system.

[0006] In a first aspect, a method for image recognition is provided, the method comprising: acquiring a first image; recognizing the first image based on a first neural network to obtain a first recognition result, the first recognition result being used to indicate that the first image includes N first signs; recognizing the first image based on a second neural network to obtain a second recognition result, the second recognition result being used to indicate that the first image includes a second sign, and the second sign is a composite sign; matching the first sign with the second sign respectively to determine that the second sign includes M first signs of the N first signs; and fusing the sign meanings corresponding to the M first signs respectively to determine the sign meaning corresponding to the second sign.

[0007] It should be understood that the above-mentioned M is greater than 1 and less than or equal to N.

[0008] For example, the first image may be taken by the camera device in the above embodiment, and the image may be an image of the surrounding environment of the road on which the vehicle is traveling.

[0009] For example, the first sign may be a conventional traffic sign proposed in the above embodiment, or a basic pattern of a traffic light, such as a red light, a green light, a yellow light, a turn signal, a countdown second, etc.

[0010] For example, the above-mentioned second sign can be a composite traffic sign proposed in the above-mentioned embodiment, for example, a vehicle type sign that is an auxiliary sign among conventional traffic signs and a speed limit sign that is a main sign among conventional traffic signs are combined; it can also be a composite traffic signal of a traffic light, for example, a green light that can be used as a main sign and a countdown second that can be used as an auxiliary sign are combined, or a green light that can be used as a main sign is combined with a green left turn light that can be used as an auxiliary sign.

[0011] For example, the first marker included in the first image can be matched with the second marker respectively by calculating the intersection over union (IOU). Before matching, a reasonable IOU threshold needs to be preset. When the IOU obtained by the marker matching is greater than or equal to the IOU threshold, it is considered that the first marker matched with the second marker is matched successfully. If it is determined that the first marker and the second marker are matched successfully, it is considered that the first marker is a component of the second marker.

[0012] For example, the IOU threshold may be adjusted according to the number M of first markers included in the second marker, and may be adjusted according to the ratio of the geometric parameter of the first marker to the geometric parameter of the second marker.

[0013] For example, the aforementioned geometric parameters may be the length and width of the logo.

[0014] Based on the above technical solution, the specific meaning of traffic signs in the road environment can be accurately identified, especially the meaning of composite traffic signs, which helps to improve the safety and reliability of the autonomous driving system or the assisted driving system. In addition, since there are many patterns of the second sign, if the second neural network is used to directly identify the specific meaning of the second sign, it will take up huge computing resources and storage resources. Therefore, the second neural network in this solution does not directly identify the specific meaning of the second sign, but only recognizes that the first image includes the second sign, and then determines the specific meaning of the second sign through subsequent sign matching and meaning fusion, thereby saving computing resources and storage resources. In addition, the second neural network is relatively simple, has high computational efficiency, and is easy to implement.

[0015] In combination with the first aspect, in some implementations of the first aspect, the first recognition result includes the meanings of the signs corresponding to the N first signs respectively.

[0016] In combination with the first aspect, in some implementations of the first aspect, the first recognition result further includes first position information corresponding to the N first marks respectively; and the second recognition result includes second position information corresponding to the second mark.

[0017] For example, the first position information may be a set of coordinates corresponding to the N first markers in a coordinate system established based on the first image, and the second position information may be a set of coordinates corresponding to the second marker in a coordinate system established based on the first image.

[0018] Based on the above technical solution, the position information of the second mark in the first image can be further determined, thereby helping to improve the accuracy of subsequent matching of the first mark and the second mark.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned composite sign is composed of a main sign and an auxiliary sign, or is composed of multiple main signs, wherein the main sign includes at least one of the following: a prohibition sign, an instruction sign, a warning sign, and a construction sign; the auxiliary sign is a sign used to indicate time, or to indicate the type of vehicle, or to indicate the area distance, or to indicate the reason for the warning and prohibition.

[0020] In combination with the first aspect, in certain implementations of the first aspect, when it is determined that the above-mentioned N is less than or equal to 1, or when it is determined that the above-mentioned first image does not include the second mark, the above-mentioned first recognition result is directly output.

[0021] For example, if the first recognition result is determined to be that the first image includes one first mark or does not include the first mark, that is, the above-mentioned N is less than or equal to 1, then the subsequent processing flow can be stopped and the above-mentioned first recognition result can be directly output. If the second recognition result is determined to be that the first image does not include the second mark, then the subsequent processing flow can be stopped and the above-mentioned first recognition result can be directly output.

[0022] Based on the above technical solution, relatively redundant processing procedures can be avoided, the processing speed of traffic sign recognition is further accelerated, and the recognition efficiency is improved.

[0023] In combination with the first aspect, in certain implementations of the first aspect, based on the second position information, it is determined that the second mark is located in the first area of ​​the first image; based on the first position information, the first mark located in the first area is filtered out; and the first mark located in the first area is matched with the second mark.

[0024] For example, if the number of second marks included in the first image is L, where L is a positive integer, the above mark matching process needs to be performed L times to determine the composition of all second marks.

[0025] Based on the above technical solution, while being able to accurately identify the basic composition of the second mark and ensure the accuracy of the subsequent determination of the corresponding meaning of the second mark, it is only necessary to match the first mark located in the first area where the second mark is located with the second mark respectively, thereby avoiding matching all the first marks with the second marks, saving computing resources, and increasing the efficiency of image recognition.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the meaning of the sign corresponding to the second sign is applied to decision-making, and the decision-making includes planning the vehicle's driving route, driving speed, or braking strategy.

[0027] Based on this technical solution, the meaning of the first and second signs obtained through image recognition can be transmitted to downstream devices, allowing them to more comprehensively determine the current road traffic regulations and formulate reasonable vehicle control strategies. This ensures the safety and reliability of the autonomous driving system's vehicle control.

[0028] In combination with the first aspect, in some implementations of the first aspect, the first neural network is a YOLO (you only look once) series model or a region-based convolutional neural network (RCNN) series model; the second neural network is a YOLO series model or an RCNN series model.

[0029] Based on the above technical solution, the first image is recognized based on the model, and the first mark and the second mark in the first image can be recognized more accurately.

[0030] In combination with the first aspect, in some implementations of the first aspect, before the first image is recognized, a preprocessing operation is performed on the first image, where the preprocessing operation includes normalization or scaling.

[0031] Based on the above technical solution, it helps to improve the accuracy of first image recognition.

[0032] In a second aspect, an image recognition device is provided, which includes: an acquisition unit for acquiring a first image; a processing unit for recognizing the first image based on a first neural network to obtain a first recognition result, wherein the first recognition result is used to indicate that the first image includes N first signs; recognizing the first image based on a second neural network to obtain a second recognition result, wherein the second recognition result is used to indicate that the first image includes a second sign, which is a composite sign; matching the first sign with the second sign respectively to determine that the second sign includes M first signs of the N first signs; fusing the sign meanings corresponding to the M first signs respectively to determine the sign meaning corresponding to the second sign.

[0033] Based on the above technical solution, the specific meaning of traffic signs in the road environment can be accurately identified, especially the meaning of composite traffic signs, which helps to improve the safety and reliability of the autonomous driving system or the assisted driving system. In addition, since there are many patterns of the second sign, if the second neural network is used to directly identify the specific meaning of the second sign, it will take up huge computing resources and storage resources. Therefore, the second neural network in this solution does not directly identify the specific meaning of the second sign, but only recognizes that the first image includes the second sign, and then determines the specific meaning of the second sign through subsequent sign matching and meaning fusion, thereby saving computing resources and storage resources. In addition, the second neural network is relatively simple, has high computational efficiency, and is easy to implement.

[0034] In combination with the second aspect, in certain implementations of the second aspect, the first recognition result includes the meanings of the signs corresponding to the N first signs respectively.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, the first recognition result further includes first position information corresponding to the N first marks; the second recognition result includes second position information corresponding to the second mark.

[0036] Based on the above technical solution, the position information of the second mark in the first image can be further determined, thereby helping to improve the accuracy of subsequent matching of the first mark and the second mark.

[0037] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned composite sign is composed of a main sign and an auxiliary sign, or is composed of multiple main signs, wherein the above-mentioned main sign includes at least one of the following: a prohibition sign, an instruction sign, a warning sign, and a construction sign; the above-mentioned auxiliary sign is a sign used to indicate time, or to indicate the type of vehicle, or to indicate the area distance, or to indicate the reason for the warning and prohibition.

[0038] In combination with the second aspect, in some implementations of the second aspect, the processing unit is further configured to: directly output the first recognition result when it is determined that N is less than or equal to 1, or when it is determined that the first image does not include the second mark.

[0039] Based on the above technical solution, relatively redundant processing procedures can be avoided, the processing speed of traffic sign recognition is further accelerated, and the recognition efficiency is improved.

[0040] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing unit is specifically used to: determine, based on the second position information, that the second mark is located in the first area of ​​the first image; based on the first position information, filter out the first mark located in the first area; and match the first mark located in the first area with the second mark.

[0041] Based on the above technical solution, while being able to accurately identify the basic composition of the second mark and ensure the accuracy of the subsequent determination of the corresponding meaning of the second mark, it is only necessary to match the first mark located in the first area where the second mark is located with the second mark respectively, thereby avoiding matching all the first marks with the second marks, saving computing resources, and increasing the efficiency of image recognition.

[0042] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing unit is also used to: apply the sign meaning corresponding to the second sign to decision-making and judgment, and the decision-making and judgment include planning the vehicle's driving route, driving speed or braking strategy.

[0043] Based on this technical solution, the meaning of the first and second signs obtained through image recognition can be transmitted to downstream devices, allowing them to more comprehensively determine the current road traffic regulations and formulate reasonable vehicle control strategies. This ensures the safety and reliability of the autonomous driving system's vehicle control.

[0044] In combination with the second aspect, in some implementations of the second aspect, the first neural network is a YOLO series model or an RCNN series model; the second neural network is a YOLO series model or an RCNN series model.

[0045] Based on the above technical solution, the first image is recognized based on the model, and the first mark and the second mark in the first image can be recognized more accurately.

[0046] In combination with the second aspect, in some implementations of the second aspect, the processing unit is further configured to: perform a preprocessing operation on the first image before recognizing the first image, where the preprocessing operation includes normalization or scaling.

[0047] Based on the above technical solution, it helps to improve the accuracy of first image recognition.

[0048] In a third aspect, an image recognition device is provided, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute a method in any possible implementation of the method design of the first aspect above.

[0049] In a fourth aspect, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, and the signals include computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method in any possible implementation of the method design of the first aspect above.

[0050] In a fifth aspect, a computer-readable storage medium is provided, storing a computer program or instruction, which is used to implement the method in any possible implementation manner in the method design of the first aspect.

[0051] In a sixth aspect, a computer program product is provided. When the computer program code or instructions are executed on a computer, the computer executes a method in any possible implementation of the method design of the first aspect.

[0052] In a seventh aspect, an embodiment of the present application provides a vehicle, which includes a device as in any possible implementation of the second aspect and the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] FIG1 is a functional schematic block diagram of a vehicle 100 provided in an embodiment of the present application.

[0054] FIG2 is an example diagram of a composite traffic sign provided in an embodiment of the present application.

[0055] FIG3 is an example diagram of another composite traffic sign provided in an embodiment of the present application.

[0056] FIG4 is an example diagram of another composite traffic sign provided in an embodiment of the present application.

[0057] FIG5 is an example diagram of another composite traffic sign provided in an embodiment of the present application.

[0058] FIG6 is a schematic diagram of an image recognition system 600 provided in an embodiment of the present application.

[0059] FIG7 is a flow chart of an image recognition method provided in an embodiment of the present application.

[0060] FIG8 is a schematic block diagram of a traffic sign recognition module provided in an embodiment of the present application.

[0061] FIG9 is a schematic diagram of a first recognition result provided in an embodiment of the present application.

[0062] FIG10 is a schematic diagram of a second recognition result provided in an embodiment of the present application.

[0063] FIG11 is a schematic diagram of a logo matching provided in an embodiment of the present application.

[0064] FIG12 is a schematic diagram of yet another type of logo matching provided in an embodiment of the present application.

[0065] FIG13 is a schematic diagram of determining the meaning of a second mark provided in an embodiment of the present application.

[0066] FIG14 is a schematic block diagram of an image recognition device 1400 provided in an embodiment of the present application.

[0067] FIG15 is a schematic block diagram of a computer device provided in an embodiment of the present application.

[0068] FIG16 is a schematic block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0070] FIG1 is a functional schematic block diagram of a vehicle 100 provided in an embodiment of the present application.

[0071] The vehicle 100 may include a perception system 120, a display device 130, and a computing platform 150. The perception system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a positioning system, which may be a global positioning system (GPS), a Beidou system or other positioning systems, an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0072] Some or all functions of the vehicle 100 may be controlled by a computing platform 150. The computing platform 150 may include processors 151 to 15n (n is a positive integer). A processor is a circuit capable of processing signals. In one implementation, the processor may be a circuit capable of reading and executing instructions, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor may implement certain functions through the logical relationships of a hardware circuit. The logical relationships of the hardware circuit may be fixed or reconfigurable. For example, the processor may be a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration file to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. In addition, the computing platform 150 can also include a memory for storing instructions, and some or all of the processors 151 to 15n can call the instructions in the memory to implement corresponding functions.

[0073] The above-mentioned vehicle 100 used in the embodiment of the present application can be an intelligent vehicle, an unmanned vehicle, an automatic driving vehicle, a vehicle equipped with relevant intelligent vehicle-mounted equipment, or a vehicle with assisted driving function, and the embodiment of the present application does not limit this.

[0074] It should be understood that the technical solutions in the embodiments of the present application can be applied to autonomous driving systems to take over the user's control of the vehicle; they can also be applied to assisted driving systems to provide users with real-time traffic information on the road and prompt users to avoid violating traffic rules.

[0075] In some possible embodiments, traffic signs defined by current national standards are conventional traffic signs, including prohibition signs, instruction signs, warning signs, auxiliary signs, and construction signs. Auxiliary signs are used to indicate time, vehicle type, distance, or the reason for a warning or prohibition. Among conventional traffic signs, signs other than auxiliary signs are defined as primary signs. These auxiliary signs cannot be set up and used independently and are usually attached near the primary sign to provide auxiliary explanations for the primary sign and limit its scope of application. Based on this, the present application defines a composite traffic sign, which consists of at least one primary sign and at least one auxiliary sign, or multiple primary signs. Typically, a composite traffic sign consists of one primary sign and at least one auxiliary sign, but there are some special cases where a composite traffic sign consists of multiple primary signs and at least one auxiliary sign, such as speed limit signs on highways, which include minimum speed limit, maximum speed limit, and vehicle type. Furthermore, a composite traffic sign may consist of only two (minimum speed limit, maximum speed limit) or three (lane information: left turn, straight ahead, right turn) primary signs.

[0076] Figure 2 shows an example of a composite traffic sign provided by an embodiment of the present application. The traffic sign shown in Figure 2 includes three composite traffic signs for indicating speed limits. Each of the three composite traffic signs is composed of a vehicle type symbol and a prohibition symbol for indicating speed limits. The three composite traffic signs indicate that the speed limit on this road is 120 km / h for small cars, 100 km / h for passenger cars, and 100 km / h for trucks, respectively.

[0077] Figure 3 shows another example of a composite traffic sign provided by an embodiment of the present application. The traffic sign shown in Figure 3 includes a composite traffic sign for indicating a speed limit. The composite traffic sign is composed of an auxiliary sign indicating the reason for the prohibition and a prohibition sign indicating the speed limit. The composite traffic sign indicates that the road ahead is a ramp and the speed limit is 40 km / h.

[0078] Figure 4 shows another example of a composite traffic sign provided by an embodiment of the present application. The traffic sign shown in Figure 4 includes a composite traffic sign for road indication, which is composed of a time sign and a prohibition sign. It indicates that U-turns are prohibited on this road between 7:00 AM and 9:00 AM, and between 3:00 PM and 5:00 PM.

[0079] Figure 5 shows another example of a composite traffic sign provided by an embodiment of the present application. The traffic sign shown in Figure 5 includes a composite traffic sign for road indication, which is composed of two prohibition signs, indicating that the speed limit for vehicles on the road is no more than 80 km / h and no less than 60 km / h.

[0080] It can be seen that the pattern structure of composite traffic signs is complex, and the corresponding meaning is more complex than that of conventional traffic signs. Moreover, through the combination of different conventional traffic signs, there are also many types and numbers of composite traffic signs.

[0081] However, if all traffic signs in a traffic environment are simply identified as such, it is impossible to accurately extract the traffic rules expressed by the composite signs. For example, the precise meaning of a speed limit sign with auxiliary signs cannot be determined. This results in information fragmentation, which in turn reduces the safety and reliability of the vehicle's autonomous or assisted driving systems. Furthermore, storing and recording the meaning of each composite traffic sign is difficult due to the numerous combinations and types of composite traffic signs.

[0082] In view of this, an embodiment of the present application proposes an image recognition method that, by identifying an acquired image, can screen out composite traffic signs and conventional traffic signs included in the current image, then match the conventional traffic signs with the composite traffic signs to determine the multiple conventional traffic signs that constitute the composite traffic sign, and then fuse the meanings corresponding to the multiple conventional traffic signs to determine the meaning of the composite traffic sign. This method can accurately identify the meaning of traffic signs in a road environment, especially the meaning of composite traffic signs, thereby helping to improve the safety and reliability of autonomous driving systems or assisted driving systems.

[0083] 6 is a schematic diagram of an image recognition system 600 applicable to an embodiment of the present application. The system can be used to recognize an image to determine the specific meaning of a traffic sign in the image.

[0084] The image recognition system 600 applicable to the embodiment of the present application includes: a camera device and an environment perception system. The camera device can be connected to the environment perception system.

[0085] Taking the above-mentioned image recognition system 600 as an example, the above-mentioned camera device may be an industrial camera, which is mainly used to capture images of the road the vehicle is currently traveling on and send the captured images to the environment perception system for corresponding processing.

[0086] The above-mentioned environmental perception system can include multiple recognition modules, and for the application scenario of traffic sign recognition, it specifically includes a traffic sign recognition module, which is used to identify traffic signs in images, and then determine the specific meanings corresponding to the traffic signs, including the specific meanings of conventional traffic signs and compound traffic signs, and finally output the traffic signs and their corresponding specific meanings.

[0087] Based on the above-provided system, the present application embodiment proposes a method for image recognition. The following will use the smart car driving scene as an example to explain the method in detail.

[0088] FIG7 shows a flow chart of an image recognition method provided in an embodiment of the present application.

[0089] S710: Acquire a first image.

[0090] In some possible embodiments, the first image may be captured by the camera device in the above embodiment, and the image may be an image of the surrounding environment of the road on which the vehicle is traveling.

[0091] In some possible embodiments, after acquiring the first image, a preprocessing operation may be performed on the first image. The preprocessing operation may include normalization, scaling, image flipping, image rotation, Gaussian filtering, etc.

[0092] S720: Based on the first neural network, recognize the first image to obtain a first recognition result, where the first recognition result is used to indicate that the first image includes N first marks.

[0093] In some possible embodiments, the first recognition result includes the meanings of the N first signs respectively corresponding to the above-mentioned N first signs.

[0094] In some possible embodiments, the first sign may be a conventional traffic sign proposed in the above embodiments, or a basic pattern of a traffic light, such as a red light, a green light, a yellow light, a turn signal, a countdown, etc.

[0095] In some possible embodiments, when the first sign is a conventional traffic sign, the first sign includes a primary sign, such as a prohibition sign, an instruction sign, a warning sign, an auxiliary sign, a construction sign, etc. In addition, the first sign also includes auxiliary signs, such as signs indicating time, vehicle type, area distance, or reasons for warnings and prohibitions. The composite traffic signs in the above embodiments are generally a combination of auxiliary signs and basic traffic signs.

[0096] S730: Recognize the first image based on a second neural network to obtain a second recognition result, where the second recognition result is used to indicate that the first image includes a second mark, and the second mark is a composite mark.

[0097] In some possible embodiments, the composite sign is composed of a main sign and an auxiliary sign, or is composed of multiple main signs. The composite sign can be a composite traffic sign proposed in the above embodiments, for example, a vehicle type sign that is an auxiliary sign among conventional traffic signs combined with a speed limit sign that is a main sign among conventional traffic signs; or it can be a composite traffic signal of a traffic light, for example, a green light that can serve as a main sign combined with a countdown that can serve as an auxiliary sign, or a green light that can serve as a main sign combined with a green left turn light that can serve as an auxiliary sign.

[0098] S740: Match the first flags with the second flags respectively to determine that the second flags include M first flags among the N first flags.

[0099] It should be understood that the above-mentioned M is greater than 1 and less than or equal to N.

[0100] It should be understood that, since all first signs included in the first image have already been identified in S720, these first signs include both standalone regular traffic signs and signs that form part of a composite traffic sign. In other words, of the N first signs in the first image, some are standalone regular traffic signs, while others are regular traffic signs that form part of a composite traffic sign. Therefore, the M first signs included in the second sign mentioned in S740 are also part of the N first signs mentioned in S720.

[0101] In some possible embodiments, the first marker included in the first image can be matched with the second marker by calculating the IOU. Before matching, a reasonable IOU threshold, such as 0.4 or 0.5, needs to be preset. When the IOU obtained by the marker matching is greater than or equal to the IOU threshold, the first marker is considered to be matched with the second marker successfully. If it is determined that the first marker and the second marker are matched successfully, the first marker is considered to be a component of the second marker.

[0102] In some possible embodiments, the IOU threshold may be adjusted according to the number M of first markers included in the second marker, and may be adjusted according to the ratio of the geometric parameter of the first marker to the geometric parameter of the second marker.

[0103] S750: The symbol meanings corresponding to the M first symbols are merged to determine the symbol meaning corresponding to the second symbol.

[0104] It should be understood that in S740, it can be determined that the second sign includes M of the N first signs. Since the M first signs each correspond to a specific meaning in S720, the meaning of the second sign can be determined by fusing the meanings of the M first signs. For example, assuming that the second sign includes two first signs, one of which is an auxiliary sign indicating that the restricted vehicle type is a sedan, and the other is a basic traffic sign indicating that the road speed limit is 120 km / h, after fusing the meanings of the two first signs, it can be determined that the meaning of the second sign is that the current road has a speed limit of 120 km / h for sedans.

[0105] Based on the above technical solution, the specific meaning of traffic signs in the road environment can be accurately identified, especially the meaning of composite traffic signs, which helps to improve the safety and reliability of the autonomous driving system or the assisted driving system. In addition, since there are many patterns of the second sign, if the second neural network is used to directly identify the specific meaning of the second sign, it will take up huge computing resources and storage resources. Therefore, the second neural network in this solution does not directly identify the specific meaning of the second sign, but only recognizes that the first image includes the second sign, and then determines the specific meaning of the second sign through subsequent sign matching and meaning fusion, thereby saving computing resources and storage resources. In addition, the second neural network is relatively simple, has high computational efficiency, and is easy to implement.

[0106] In some possible embodiments, the above S720 and S730 may be processed in parallel or performed sequentially.

[0107] In some possible embodiments, during the execution of S720, if the first recognition result is determined to be that the first image includes one first mark or does not include the first mark, that is, the N is less than or equal to 1, then the subsequent processing flow of S730 to S750 may be stopped, and the first recognition result may be directly output, such as one first mark and its meaning, or a null value (NULL). If the second recognition result is determined to be that the first image does not include the second mark, then the subsequent processing flow of S740 and S750 may be stopped, and the first recognition result may be directly output, such as N first marks and their meanings.

[0108] Based on the above technical solution, relatively redundant processing procedures can be avoided, the processing speed of traffic sign recognition is further accelerated, and the recognition efficiency is improved.

[0109] In some possible embodiments, during the execution of S720, when it is determined that the first recognition result is that the first image includes one first mark or does not include the first mark, S730 may be continued to be executed. Through the recognition process of S730, the accuracy of the first recognition result of S720 may be reversely verified:

[0110] For example, if N is determined to be 1 in S720, and if the second mark is also determined not to be present in the first image in S730, then the recognition result in S720 is accurate, and S740 and S750 need not be executed, and the first recognition result is directly output. However, if the second mark is determined to be present in the first image in S730, then the recognition result in S720 is incorrect, and S720 may be re-executed. Based on the re-determined first recognition result, the need to continue with S740 and S750 may be determined.

[0111] Similarly, the recognition result of S720 can also reversely verify whether the recognition result of S730 is accurate.

[0112] Based on the above technical solution, it helps to improve the accuracy of image recognition of the first neural network and the second neural network.

[0113] In some possible embodiments, the first neural network may be a pre-trained target detection model, and further may be a YOLO series model or an RCNN series model. The first neural network needs to be pre-trained to be able to recognize all first landmarks included in the first image and extract each first landmark. The specific training process of the first neural network is as follows:

[0114] Taking the first image as an image of the traffic environment around the road as an example, it is first necessary to determine at least one training sample, which includes a training image and a sample label of the training sample. The training image includes various first signs, that is, various conventional traffic signs. The sample label of the training sample is used to indicate the sign meanings corresponding to the various first signs, for example, the meanings of signs such as "no passage" and "speed limit" as main signs, and the meanings of auxiliary text descriptions as auxiliary signs. Then, based on the above at least one training sample, the first neural network is trained so that in subsequent practical applications, after obtaining the first sign, the first neural network can extract the first sign and determine the sign meaning corresponding to the first sign. In addition, on this basis, the first neural network can also be trained to determine the geometric parameters of the first sign and the first position information of the first sign in the entire first image while determining the sign meaning of the first sign.

[0115] In some possible embodiments, based on the first neural network trained by the method corresponding to the above embodiment, the first recognition result determined by recognizing the first image through the first neural network may include the meanings of the signs corresponding to the N first signs in the first image.

[0116] In some possible embodiments, the first recognition result may also include geometric parameters of the N first markers, for example, the length and width of each of the N first markers; it may also include first position information of the N first markers, for example, the coordinate sets corresponding to the N first markers in the coordinate system established based on the first image.

[0117] Based on the above technical solution, the geometric parameters and position information of the first mark in the first image can be further determined, thereby helping to improve the accuracy of subsequent second mark recognition.

[0118] In some possible embodiments, the second neural network may be a pre-trained target detection model, and further may be a YOLO series model or an RCNN series model. Similar to the first neural network, the second neural network also needs to be pre-trained to be able to recognize whether the first image includes the second marker and extract the second marker. The specific training process of the second neural network is as follows:

[0119] Taking the example of a first image representing the traffic environment surrounding a road, at least one training sample must first be determined. This training sample includes a training image and a sample label for the training sample. The training image includes various second landmarks, i.e., various second landmarks formed by combining the aforementioned primary landmarks and auxiliary landmarks, such as a second landmark formed by combining a primary landmark and at least one auxiliary landmark. A second neural network is then trained based on this at least one training sample, so that in subsequent practical applications, upon acquiring the first image, the second neural network can determine that the first image includes the second landmark and extract the second landmark. Furthermore, based on this, the second neural network can also be trained to determine the geometric parameters of the second landmark and its second position information within the first image.

[0120] In some possible embodiments, since the second sign may be a composite traffic sign and there are a large number of them, it is difficult to store the relevant information of all composite traffic signs. Therefore, unlike the first recognition result, the second recognition result only includes the category of specifying that the traffic sign is a second sign, that is, determining that the first image includes the second sign.

[0121] In some possible embodiments, the second recognition result may also include geometric parameters of the second mark, such as the length and width of the second mark; it may also include second position information of the second mark, such as the coordinate set corresponding to the second mark in the coordinate system established based on the first image.

[0122] Based on the above technical solution, the geometric parameters and position information of the second mark in the first image can be further determined, thereby helping to improve the accuracy of subsequent matching of the first mark and the second mark.

[0123] In some possible embodiments, the first neural network and the second neural network may be two models of different types, for example, the first neural network is a YOLO series model, and the second neural network is an RCNN series model; or they may be two models of the same type, for example, the first neural network and the second neural network are both YOLO series models or RCNN series models.

[0124] In some possible embodiments, the specific functions of the first neural network and the second neural network can also be completed by a model, such as a third model. That is, a third model can be used to detect the first mark and the second mark in the first image respectively, and perform the related actions of S720 and S730 mentioned above.

[0125] In some possible embodiments, after training the second neural network, the second neural network can determine different configurations of the second marker, such as a combination of one primary marker and one auxiliary marker, or a combination of multiple primary markers. Furthermore, the primary marker and the auxiliary markers in the second marker are positioned adjacent or close to each other; if the primary marker and the auxiliary marker are far apart, they cannot be combined. The determination of this basic information can be achieved based on the trained second neural network, and corresponding second markers can be determined for different configurations of the basic information.

[0126] In some possible embodiments, in the autonomous driving system 600 proposed in the above embodiments, the above S720 to S750 can be implemented by the traffic sign recognition module in the autonomous driving system 600, and specifically can be jointly implemented by multiple sub-modules included in the traffic sign recognition module.

[0127] FIG8 shows a schematic block diagram of a traffic sign recognition module provided in an embodiment of the present application.

[0128] In some possible embodiments, the multiple sub-modules may include: an image acquisition module, a first recognition module, a second recognition module, a matching module, a fusion module, and a result output module.

[0129] The image acquisition module may be used to implement the image preprocessing operation and image data augmentation operation mentioned in the above embodiments.

[0130] The first recognition module is a module that stores the above-mentioned first neural network, and is mainly used to implement the above-mentioned S720. It can at least recognize all the first signs included in the first image and their corresponding sign meanings to obtain a first recognition result.

[0131] FIG9 shows a schematic diagram of a first recognition result proposed in an embodiment of the present application.

[0132] The marks in the dotted boxes shown in FIG9 are all recognized first marks.

[0133] The second recognition module is a module that stores the second neural network, and is mainly used to implement the above S730. It can at least recognize the second mark included in the first image and obtain a second recognition result.

[0134] FIG10 shows a schematic diagram of a second recognition result proposed in an embodiment of the present application.

[0135] Among them, the marks in the dotted box shown in FIG10 are all recognized second marks.

[0136] The matching module is mainly used to implement the above S740 to determine the composition of the above second mark.

[0137] In some possible embodiments, the above S740 may be implemented in the following manner.

[0138] Method 1: Match the N first markers included in the first image with the second markers respectively. The IOU calculation method proposed in the above embodiment can be used to determine which of the N first markers are components of the second marker.

[0139] FIG11 shows a schematic diagram of a logo matching proposed in an embodiment of the present application.

[0140] The first recognition module identified the traffic sign as consisting of six first signs. The second recognition module identified three second signs, of which only one, designated as second sign a, is shown in Figure 11. Based on Method 1 above, the six first signs were matched against the patterns of second sign a. Only the auxiliary sign restricting passenger car models and the primary sign indicating a speed limit of 120 km / h successfully matched second sign a. Therefore, it was determined that second sign a was composed of the auxiliary sign restricting passenger car models and the primary sign indicating a speed limit of 120 km / h.

[0141] Based on the above technical solution, the basic composition of the second mark can be accurately identified, thereby ensuring the accuracy of the subsequent determination of the corresponding meaning of the second mark.

[0142] Method 2: When the first recognition result includes first position information and second position information, wherein the first position information is used to indicate the positions of the N first markers in the first image, and the second position information is used to indicate the position of the second marker in the first image, the second marker can be determined to be located in the first area of ​​the first image based on the second position information. Then, based on the first position information, the first marker located in the first area is screened out. Finally, the first marker located in the first area is matched with the second marker. The first marker located in the first area can be matched with the second marker using the IOU calculation method proposed in the above embodiment to determine which of the first markers in the first area are components of the second marker.

[0143] It should be understood that if the number of second marks included in the first image is L, where L is a positive integer, the above mark matching process needs to be performed L times to determine the composition of all second marks.

[0144] FIG12 shows a schematic diagram of another type of landmark matching proposed in an embodiment of the present application.

[0145] Based on the first recognition module, it was identified that the traffic sign includes 6 first signs. Based on the second recognition module, 3 second signs were identified. Take the second sign a shown in Figure 12 as an example for illustration: Based on the above method 2, the first area where the second sign a is located is determined, and then it is determined that the first sign located in the first area is an auxiliary sign that restricts car models and a main sign with a speed limit of 120km / h. Then, these two first signs are matched with the second sign a respectively. The result is a successful match, so it is finally determined that the second sign a is composed of the above-mentioned auxiliary sign that restricts car models and the above-mentioned main sign with a speed limit of 120km / h.

[0146] Based on the above technical solution, while being able to accurately identify the basic composition of the second mark and ensure the accuracy of the subsequent determination of the corresponding meaning of the second mark, it is only necessary to match the first mark located in the first area where the second mark is located with the second mark respectively, thereby avoiding matching all the first marks with the second marks, saving computing resources, and increasing the efficiency of image recognition.

[0147] The above fusion module is mainly used to implement the above S750 to determine the meaning of the sign corresponding to the second sign.

[0148] In some possible embodiments, before determining the meaning of the sign corresponding to the second sign, it is possible to first determine which signs in the second sign are main signs and which signs are auxiliary signs, and based on the corresponding grammatical form, the meaning of the auxiliary sign can be used as a qualifier of the meaning of the main sign, so that the meaning of the sign corresponding to the second sign can be accurately determined.

[0149] FIG13 shows a schematic diagram of determining the meaning of the second mark proposed in an embodiment of the present application.

[0150] Among them, there are three second signs. Based on the above method, the meanings of these three second signs are as follows: the speed limit for cars is 120km / h, the speed limit for buses is 100km / h, and the speed limit for trucks is 100km / h. The recognition results of the meanings of these three second signs are then sent to the result output module.

[0151] It should be understood that when there are some independently established and used first marks, ie, main marks, in the first image, the meaning recognition results of these first marks also need to be sent to the result output module.

[0152] The result output module is mainly used to output the sign meaning corresponding to the second sign and the sign meaning corresponding to other first signs except those constituting the second sign.

[0153] Based on the above technical solution, by installing corresponding sub-modules in the autonomous driving system 600, the specific meaning of traffic signs in the road environment can be accurately identified, especially the meaning of composite traffic signs can be accurately identified, which helps to improve the safety and reliability of the autonomous driving system or assisted driving system.

[0154] In some possible embodiments, after receiving the meaning of the first sign and / or the second sign, the downstream device may make a decision based on the meaning of the first sign and / or the second sign. The decision may include planning the vehicle's route, speed, and braking strategy.

[0155] Based on this technical solution, the meaning of the first and second signs obtained through image recognition can be transmitted to downstream devices, allowing them to more comprehensively determine the current road traffic regulations and formulate reasonable vehicle control strategies. This ensures the safety and reliability of the autonomous driving system's vehicle control.

[0156] In some possible embodiments, the above method may also be applicable to scenarios where the first image only includes the first mark. For example, the first image includes at least one first mark. After recognition based on the second neural network in S730, if a third recognition result is determined, the third recognition result is used to indicate that the at least one first mark cannot constitute the second mark, that is, the first image does not include the second mark. Then, there is no need to execute S740 and S750, and the meaning of the at least one first mark can be directly output. After recognition based on the second neural network in S730, if the second recognition result is determined, it can be determined that the at least one first mark together constitutes at least one second mark, and it is necessary to execute S740 and S750 to output the meaning of the at least one second mark.

[0157] In some possible embodiments, after recognition is performed based on the first neural network in S720, if a fourth recognition result is determined, and the fourth recognition result is used to indicate that the first mark does not exist in the first image, there is no need to execute subsequent S730 to S750, and no result may be output.

[0158] In some possible embodiments, after the first image is recognized based on the first neural network and the second neural network, if the first recognition result and the third recognition result are respectively determined, there is no need to execute S740 and S750, and the meaning of the first sign corresponding to the first recognition result can be directly output.

[0159] In addition, an embodiment of the present application also provides a device for implementing any of the above methods. For example, a vehicle control device is provided, which includes a unit (or means) for implementing any of the above vehicle control methods.

[0160] FIG14 shows a schematic block diagram of an image recognition apparatus 1400 provided in an embodiment of the present application. As shown in FIG14 , the apparatus 1400 includes:

[0161] The acquisition unit 1410 is used to acquire a first image; the processing unit 1420 is used to identify the first image based on a first neural network and determine a first recognition result, wherein the first recognition result is used to indicate that the first image includes N first signs and the sign meanings corresponding to the N first signs respectively, and the first neural network is a pre-trained neural network model; based on the second neural network, the first image is identified and a second recognition result is determined, wherein the second recognition result is used to indicate that the first image includes a second sign, and the second sign is a composite sign, which is composed of a main sign and an auxiliary sign, or composed of multiple main signs, and the second neural network is a pre-trained neural network model; the first sign included in the first image is matched with the second sign respectively to determine that the second sign includes M first signs of the above-mentioned N first signs; the sign meanings corresponding to the M first signs are merged to determine the sign meaning corresponding to the second sign.

[0162] In some possible embodiments, the first recognition result further includes the first position information corresponding to the N first marks; the second recognition result includes the second position information corresponding to the second mark.

[0163] In some possible embodiments, the above-mentioned main signs include at least one of the following: prohibition signs, instruction signs, warning signs, and construction signs; the above-mentioned auxiliary signs are signs used to indicate time, or indicate vehicle types, or indicate area distances, or indicate reasons for warnings and prohibitions.

[0164] In some possible embodiments, the processing unit 1420 is further configured to: when it is determined that N is less than or equal to 1, or when it is determined that the first image does not include the second mark, directly output the first recognition result.

[0165] In some possible embodiments, the above-mentioned processing unit 1420 is specifically used to: determine that the second mark is located in the first area of ​​the first image based on the second position information; filter out the first mark located in the first area based on the first position information; and match the first mark located in the first area with the second mark.

[0166] In some possible embodiments, the processing unit 1420 is further configured to apply the meaning of the sign corresponding to the second sign to decision-making, where the decision-making includes planning the vehicle's driving route, driving speed, or braking strategy.

[0167] In some possible embodiments, the first neural network is a YOLO series model or an RCNN series model; the second neural network is a YOLO series model or an RCNN series model.

[0168] In some possible embodiments, the processing unit 1420 is further configured to: perform a preprocessing operation on the first image before recognizing the first image, where the preprocessing operation includes normalization or scaling.

[0169] FIG15 shows a schematic block diagram of another image recognition apparatus 1500 provided in an embodiment of the present application. The computer device 1500 shown in FIG15 includes a memory 1510, a processor 1520, and a bus 1540. Optionally, the computer device 1500 also includes a communication interface 1530. The memory 1510, the processor 1520, and the communication interface 1530 are connected to each other via the bus 1540.

[0170] Memory 1510 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 1510 can store programs. When the program stored in memory 1510 is executed by processor 1520, processor 1520 is configured to perform the various steps of the image recognition method of the embodiment of the present application. Specifically, processor 1520 can execute the method shown in FIG. 7 above.

[0171] The processor 1520 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the image recognition method proposed in the method embodiment of the present application.

[0172] The processor 1520 may also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the image recognition method of the present application may be completed by an integrated logic circuit of hardware in the processor 1520 or instructions in software form.

[0173] The processor 1520 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The steps of the methods involved in the embodiments of this application may be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory 1510. The processor 1520 reads the information in the memory 1510 and, in conjunction with its hardware, performs the functions required to be performed by the units included in the apparatus shown in FIG. 14 , or executes the method shown in FIG. 7 of the method embodiment of this application.

[0174] The communication interface 1530 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 1500 and other devices or a communication network. For example, training data can be obtained through the communication interface 1530 .

[0175] The bus 1540 may include a path for transmitting information between the various components of the device 1500 (eg, the memory 1510 , the processor 1520 , and the communication interface 1530 ).

[0176] It should be understood that although the above-described apparatus 1500 only illustrates a memory, a processor, and a communication interface, during specific implementation, those skilled in the art will appreciate that the apparatus 1500 may also include other components necessary for normal operation. Furthermore, those skilled in the art will appreciate that, depending on specific needs, the apparatus 1500 may also include hardware components that implement other additional functions. Furthermore, those skilled in the art will appreciate that the apparatus 1500 may include only the components necessary to implement the embodiments of the present application, and need not include all of the components shown in FIG. 15 .

[0177] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0178] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0179] Figure 16 is a schematic block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 1600 shown in Figure 16 stores computer instructions 1610. When the computer instructions 1610 are executed by a processor, the method corresponding to the above embodiment can be implemented.

[0180] In some possible embodiments, computer-readable storage medium 1600 may be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes a collection of one or more available media. Available media may be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media may be a solid-state drive.

[0181] In the embodiments of the present application, “first”, “second” and various numbers are merely distinctions for ease of description and are not intended to limit the scope of the embodiments of the present application, for example, to distinguish between different signs, models, etc.

[0182] It should be understood that the term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0183] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0184] In the embodiments of the present application, the above-mentioned embodiments can be implemented in whole or in part by software, hardware, firmware or other arbitrary combinations. When implemented using software, the above-mentioned embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When loading or executing a computer instruction or computer program on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.) mode.

[0185] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0186] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0189] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0190] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0191] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for image recognition, characterized in that: The method comprises: acquiring a first image; Recognize the first image based on a first neural network to obtain a first recognition result, where the first recognition result is used to indicate that the first image includes N first marks; Recognize the first image based on a second neural network to obtain a second recognition result, where the second recognition result is used to indicate that the first image includes a second mark, and the second mark is a composite mark; Matching the first flags with the second flags respectively to determine that the second flags include M first flags among the N first flags; The sign meanings corresponding to the M first signs are merged to determine the sign meaning corresponding to the second sign.

2. The method according to claim 1, characterized in that The first recognition result includes the sign meanings corresponding to the N first signs respectively.

3. The method according to claim 1 or 2, characterized in that: The first recognition result also includes first position information corresponding to the N first marks respectively; the second recognition result includes second position information corresponding to the second mark.

4. The method according to any one of claims 1 to 3, characterized in that The composite sign is composed of a main sign and an auxiliary sign, or is composed of multiple main signs; wherein the main sign includes at least one of the following: a prohibition sign, an instruction sign, a warning sign, and a construction sign; the auxiliary sign is a sign used to indicate time, or to indicate the type of vehicle, or to indicate the area distance, or to indicate the reason for the warning and prohibition.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: When it is determined that N is less than or equal to 1, or when it is determined that the first image does not include the second mark, the first recognition result is directly output.

6. The method according to any one of claims 3 to 5, characterized in that The matching of the first mark included in the first image with the second mark respectively comprises: determining, according to the second position information, that the second mark is located in a first area in the first image; Filtering the first mark located in the first area according to the first location information; The first mark located in the first area is matched with the second mark.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The sign meaning corresponding to the second sign is applied to decision-making, and the decision-making includes planning the vehicle's driving route, driving speed or braking strategy.

8. The method according to any one of claims 1 to 7, characterized in that The first neural network is a YOLO series model or an RCNN series model; The second neural network is a YOLO series model or an RCNN series model.

9. The method according to any one of claims 1 to 8, characterized in that Before identifying the first image, the method further includes: A preprocessing operation is performed on the first image, where the preprocessing operation includes normalization or scaling.

10. An image recognition device, characterized in that: The device comprises: An acquisition unit, configured to acquire a first image; A processing unit is used to recognize the first image based on a first neural network to obtain a first recognition result, wherein the first recognition result is used to indicate that the first image includes N first signs; recognize the first image based on a second neural network to obtain a second recognition result, wherein the second recognition result is used to indicate that the first image includes a second sign, and the second sign is a composite sign; match the first sign with the second sign respectively to determine that the second sign includes M first signs among the N first signs; and fuse the sign meanings corresponding to the M first signs respectively to determine the sign meaning corresponding to the second sign.

11. The device according to claim 10, characterized in that The first recognition result includes the sign meanings corresponding to the N first signs respectively.

12. The device according to claim 10 or 11, characterized in that The first recognition result also includes first position information corresponding to the N first marks respectively; the second recognition result includes second position information corresponding to the second mark.

13. The device according to any one of claims 10 to 12, characterized in that The composite sign is composed of a main sign and an auxiliary sign, or is composed of multiple main signs; wherein the main sign includes at least one of the following: a prohibition sign, an instruction sign, a warning sign, and a construction sign; the auxiliary sign is a sign used to indicate time, or to indicate the type of vehicle, or to indicate the area distance, or to indicate the reason for the warning and prohibition.

14. The device according to any one of claims 10 to 13, characterized in that The processing unit is also used for: When it is determined that N is less than or equal to 1, or when it is determined that the first image does not include the second mark, the first recognition result is directly output.

15. The device according to any one of claims 10 to 14, characterized in that The processing unit is specifically used for: determining, according to the second position information, that the second mark is located in a first area in the first image; Filtering the first mark located in the first area according to the first location information; The first mark located in the first area is matched with the second mark.

16. The device according to any one of claims 10 to 15, characterized in that The processing unit is also used for: The sign meaning corresponding to the second sign is applied to decision-making, and the decision-making includes planning the vehicle's driving route, driving speed or braking strategy.

17. The device according to any one of claims 10 to 16, characterized in that The first neural network is a YOLO series model or an RCNN series model; The second neural network is a YOLO series model or an RCNN series model.

18. The device according to any one of claims 10 to 17, characterized in that The processing unit is also used for: Before recognizing the first image, a preprocessing operation is performed on the first image, where the preprocessing operation includes normalization or scaling.

19. An image recognition device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the method according to any one of claims 1 to 9.

20. A chip system, characterized in that: The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method as described in any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 9.

22. A computer program product, characterized in that When the computer program code or instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 9.

23. A vehicle, characterized in that: Comprising a device as claimed in any one of claims 10 to 19.